From e9a47dbb4c8e9ac1c3654413ddb3300f73a78347 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Mon, 27 Jul 2026 21:28:21 +0500 Subject: [PATCH 01/13] docs(design-proposals): add "Cozystack as a distribution" Proposes replacing Cozystack's single build-stamped version with four: a semver core (apiserver, controller, operator, CRDs, delivery contract), independently semver-versioned packages and apps, and a monthly CalVer distribution release defined by a manifest that pins an exact, tested set. The case is made from platform release engineering rather than from ecosystem ambition. Four costs are paid every cycle today, in a repository with no external catalogs at all: a fix cannot reach a released cluster except through backport automation that fails silently on conflict; a month whose substance is three bug fixes still moves 160 chart versions; a breaking change in one app is a platform-level event, which is why the schema debt #6 wants to pay down accumulated in the first place; and release readiness is all-or-nothing, as v1.6.0's four RCs and rc.1 NO-GO showed. None of those require a marketplace. The machinery that fixes them happens to be the machinery an ecosystem needs, which sets build order rather than motivation. Written as a joining piece for work already in flight. It supplies the version axis #43 assumes for catalog tiers and its declared minimum platform version, reuses the cozypkg surface from #12 and the backend from #23, pairs package MAJOR with #6's storage-version conversion to give "breaking" a testable definition, and extends #39's "metadata belongs on the ApplicationDefinition, not in a directory name" one tier up. On repository-as-unit versus package-as-unit: #18's reasoning is adopted unchanged for community repositories, where the tested-together guarantee holds and the author owes Cozystack no compatibility. The first-party archive is treated differently for one reason only -- a repository-level version cannot express a partial upgrade. #12's package-level axis is therefore revived on release-engineering grounds rather than its original ecosystem ones, with its concrete surface reused as written. Grounded in a survey of the current delivery chain: the whole packages/ tree is one OCI artifact selected by one digest and fanned out to 98 PackageSources with no version field anywhere in the data model. Notes that the runtime is already data-driven, that bundles are already metapackages, that resource-policy=keep already gives dpkg-selections semantics, and that four packages already carry per-package pre-upgrade migration hooks the global counter could be generalised onto. Records that core is not low-churn (~620 commits/12mo across internal/, pkg/, api/) and flags the one experiment that decides the implementation: whether ExternalArtifact revision is content-derived or source-derived. Assisted-By: Claude Signed-off-by: Myasnikov Daniil --- .../cozystack-as-a-distribution/README.md | 481 ++++++++++++++++++ 1 file changed, 481 insertions(+) create mode 100644 design-proposals/cozystack-as-a-distribution/README.md diff --git a/design-proposals/cozystack-as-a-distribution/README.md b/design-proposals/cozystack-as-a-distribution/README.md new file mode 100644 index 0000000..71875dc --- /dev/null +++ b/design-proposals/cozystack-as-a-distribution/README.md @@ -0,0 +1,481 @@ +# Cozystack as a distribution + +- **Title:** `Cozystack as a distribution — semver core, independently versioned packages, and a monthly CalVer release train` +- **Author(s):** `@myasnikovdaniil` +- **Date:** `2026-07-27` +- **Status:** Draft + +## Overview + +Cozystack has exactly one version number, and everything wears it. 160 of the 164 charts under `packages/` ship with `version: 0.0.0 # Placeholder, the actual version will be automatically set during the build process`, stamped at build time from `COZYSTACK_VERSION`; the whole `packages/` tree is pushed as a single OCI artifact; a single digest in `packages/core/installer/values.yaml` selects it; and one `OCIRepository` fans that digest out to 98 `PackageSource` objects. A Postgres chart fix and a change to the aggregated apiserver are, as far as delivery is concerned, the same event. + +This proposal replaces that single number with four, arranged the way an operating-system distribution arranges its own: a **core** on semantic versioning (the apiserver, controller, operator, CRDs, and the delivery contract itself), **packages** and **apps** each carrying their own semantic version, and a **distribution** — the thing users install and talk about — on calendar versioning, released monthly. A release stops being "the state of a git tree at a tag" and becomes what a distribution release has always been: a manifest that pins an exact set of component versions, tested together, shipped together, supported together. + +The change is smaller than it sounds, because the platform is already most of the way there and nobody has been calling the pieces by their distribution names. `PackageSource` and `Package` are a package database. Bundles are metapackages. `helm.sh/resource-policy: keep` on the `Package` CRs means the installed set already diverges from the shipped set — that is `dpkg --get-selections`, implemented by accident. `cozypkg` is `apt` with the version, repository, and index code removed. The per-package pre-upgrade migration hook exists and is used by four packages already. What is genuinely missing is narrow: a versioned pool, a version field on the source reference, and an index. This proposal specifies those three and then connects the in-flight proposals that each assume some part of them. + +The case is made from platform release engineering rather than from ecosystem ambition — the costs it addresses are paid every release cycle today, in a repository with no external catalogs at all. That framing is deliberate and is set out in [Why this is a release-engineering proposal](#why-this-is-a-release-engineering-proposal-not-a-marketplace-proposal), because it determines build order: the internal requirement can be satisfied with no discovery surface whatsoever, while a discovery surface cannot be satisfied without it. + +## Scope and related proposals + +This proposal is deliberately a joining piece. Several accepted or in-flight proposals each solve one axis of the same problem and each stop at the edge of the version question; the intent here is to supply the axis they share rather than to re-litigate any of them. + +| Proposal | What it establishes | How this proposal relates | +|---|---|---| +| [community#43](https://github.com/cozystack/community/pull/43) — Out-of-tree application catalogs (`@lllamnyp`, Review) | Three application tiers (platform / curated catalog / external catalog), catalog repositories as OCI artifacts, e2e cost as the driver, a declared minimum platform version checked at `PackageSource` reconciliation | **Supplies the version axis #43 assumes.** #43 defines the *component structure* of the distribution (Debian's main/universe split); this proposal defines what a version means inside it and where the declared minimum platform version comes from. #43 is the stronger near-term motivator and should not wait on this | +| [community#18](https://github.com/cozystack/community/pull/18) — Cozymarketplace (`@kvaps`, Draft) | Repository-as-unit, a krew-style meta-index, a repository is one versioned OCI artifact, `cozypkg` repository commands | **Adopted as-is for external repositories; extended, not replaced, for the platform's own.** #18 makes per-package pinning a non-goal, which is the right call for the problem #18 is solving. This proposal reaches a different conclusion for first-party packages from a different requirement — see [§8](#8-repository-as-unit-versus-package-as-unit) | +| [community#12](https://github.com/cozystack/community/pull/12) — Community package index and `cozypkg` authoring (`@kvaps`, `@IvanHunters`, closed 2026-07-16) | `cozypkg tap` / `init` / `push` / `search`, metadata-only index, `community.` name prefixing, an optional expected-signing-identity per entry | **Its package-level axis is revived here, on a different justification.** #12 was package-centric; #18 superseded it with a repository-centric model and #12 was closed. That supersession is sound on ecosystem grounds. This proposal re-derives the package-level axis from platform release engineering instead, and reuses #12's `cozypkg` surface and index-entry shape rather than inventing new ones | +| [community#23](https://github.com/cozystack/community/pull/23) — Cozymarketplace supplementary (`@IvanHunters`, Draft) | Marketplace endpoints in `cozystack-api`, a `TapIndex` cache, `secretRef` threading for private taps, `cozypkg validate` | **Consumed unchanged.** The `TapIndex` cache is the natural home for the release manifest / index reader described below | +| [community#6](https://github.com/cozystack/community/pull/6) — ApplicationDefinition multi-version conversion (`@kvaps`, Draft) | `versions[]` with a storage version, `to`/`from` conversion templates, a `_version` stamp, a background migration controller | **Orthogonal axis, and the thing that makes semver honest.** #6 versions the *API surface* of an app; this proposal versions the *package*. Their relationship is what gives "breaking change" a testable definition — see [What MAJOR means](#what-major-means) | +| [community#39](https://github.com/cozystack/community/pull/39) — Fold `extra` into `apps` (`@myasnikovdaniil`, Draft) | Retires the `extra` bucket; visibility / cardinality / protection / capability as declarative `ApplicationDefinition` fields | **Same direction, one tier down.** #39 removes a directory-as-metadata convention by moving the metadata onto the ApplicationDefinition; this proposal does the same for the version. Together they turn ApplicationDefinition into the package's control file | +| [cozystack#3448](https://github.com/cozystack/cozystack/pull/3448) — ApplicationGroupDefinition (MVP) | Dynamic API-group registration, reserved-namespace validation, restart-based pickup via the `cozystack.io/config-hash` rollout | **Prerequisite for tier 2, and the proof that the runtime is already data-driven.** The mechanism it extends is the reason per-package delivery is feasible at all | +| [community#21](https://github.com/cozystack/community/pull/21) — Self-hosted in-cluster registry for air-gap (`@gecube`, Draft) | Offline bundle, in-cluster registry as source of truth | **Affected, and improved.** A per-package pool changes what an offline bundle contains; a release manifest is exactly the mirror list an air-gapped site needs. Flagged as an open question rather than solved here | +| [community#25](https://github.com/cozystack/community/pull/25), [community#33](https://github.com/cozystack/community/pull/33) | Per-cluster etcd; ComputePlane as an operator-owned module | Downstream consumers of the tiering; not blocked by this proposal | + +**Deliberately out of scope:** the content of any specific tier assignment (that is #43's call), the API-conversion mechanism (that is #6's), the dashboard's marketplace information architecture (#18/#23), and anything about how Talos or the installer bootstraps a cluster. + +## Context + +### What the delivery chain actually does today + +Every arrow in the chain below carries a *name*. The only version in the entire path is the digest at the very top, and there is exactly one of it. + +``` +packages/core/installer/values.yaml:18-19 platformSourceUrl + platformSourceRef: digest=sha256:… + └─ cmd/cozystack-operator/main.go:564-602 one OCIRepository/cozystack-platform in cozy-system + └─ packages/core/platform/templates/repository.yaml:19 spec cloned verbatim → OCIRepository/cozystack-packages + └─ packages/core/platform/sources/*.yaml 98 PackageSource CRs, all sourceRef.name: cozystack-packages + └─ internal/operator/packagesource_reconciler.go:261-278 one ArtifactGenerator per PackageSource + │ sources[0] copied from spec.sourceRef — no revision selector + │ copy: "@cozystack-packages//**" → "@artifact//" + └─ source-watcher v2.1.0 slices the tree into per-component ExternalArtifacts + └─ internal/operator/package_reconciler.go:79-83 HelmRelease.chartRef → ExternalArtifact + CrossNamespaceSourceReference for ExternalArtifact carries no version field +``` + +The pool is built by `packages/core/installer/Makefile:37`, which does `flux push artifact oci://$(REGISTRY)/cozystack-packages:$(IMAGE_TAG) --path=../../../packages` — the entire `packages/` tree becomes one OCI artifact — and then writes the resulting digest back into `values.yaml` with `yq`. + +There is no version field anywhere in the data model. `PackageSourceRef` (`api/v1alpha1/packagesource_types.go:97-116`) is `{Kind, Name, Namespace, Path}`. `Component` (`:168-189`) is `{Name, Path, Install, Libraries, ValuesFiles}`. `Package` (`api/v1alpha1/package_types.go:54-100`) is `{Variant, IgnoreDependencies, Components{Enabled, Values}}`. `path: apps/postgres` is a directory inside an immutable tarball, not a coordinate in a version space. The same path appears six times across the networking variants precisely *because* it always resolves to the same bytes — a property that per-component versioning would turn from a convenience into a hazard. + +### Core is not low-churn + +A natural instinct is that extracting core is easy because core rarely changes. Measured over the last twelve months on `main`, that is not true — the `cmd/` entrypoints are thin wrappers and the substance lives elsewhere: + +| Area | Commits (12 months) | +|---|---| +| `cmd/cozystack-api` | 1 | +| `cmd/cozystack-controller` | 25 | +| `cmd/cozystack-operator` | 27 | +| `internal/` | 284 — of which `internal/controller` 125, `internal/backupcontroller` 53, `internal/operator` 34 | +| `pkg/` | 168 — of which `pkg/registry` 115 | +| `api/` | 171 — of which `api/apps` 95 | +| `packages/apps` | 768 | +| `packages/system` | 1345 | +| `packages/core/platform` | 337 | +| `packages/core/installer` | 112 | + +Core as a body of code is roughly 620 commits a year — the second-most-churned area in the repository. This does not argue against the split; it argues that core will cut releases *more often than monthly*, which the design must accommodate rather than assume away. The consequence is spelled out in [Core releases between trains](#core-releases-between-trains). + +### The runtime is already data-driven + +The single most encouraging fact for this proposal: an app's served API is not compiled into the apiserver. `pkg/cmd/server/start.go` lists `ApplicationDefinition` objects at boot and registers served resources from them, with the OpenAPI schema carried as a string in `spec.application.openAPISchema` inside the package itself (`packages/system/-rd/cozyrds/.yaml`). `internal/controller/applicationdefinition_controller.go` watches those objects and rolls the `cozystack-api` DaemonSet by updating a config-hash on the pod template — the same mechanism cozystack#3448 extends for API groups. + +So a package can already change its own API surface without a core rebuild. That property is what makes independent package versioning feasible at all, and it is shipping today. + +One counter-fact to handle: `api/apps/v1alpha1/` is a **separate Go module** (`api/apps/v1alpha1/go.mod`) tagged in lockstep with the platform (`api/apps/v1alpha1/v1.6.0`, `/v1.5.3`, `/v1.4.6`), mirroring app schemas for external consumers — and `internal/backupcontroller/*app/types.go` duplicates several of them again. Under per-app versioning that module's version stops meaning anything. + +### The problem + +- **Every release moves every chart.** Because chart versions are stamped from `COZYSTACK_VERSION`, a release changes the version of 160 charts whether or not their content changed. Users upgrading for a Postgres fix take the whole platform with it. Upgrade windows, blast radius, and rollback all scale with the size of the tree rather than the size of the change. +- **Breaking changes are held hostage.** A breaking change in one app forces either a platform major (which frightens users away from an upgrade that is mostly bug fixes) or an indefinite delay. There is no way to say "MongoDB 4.0 is breaking, the platform is not". +- **A package fix cannot ship without a platform release.** Today the only route to a released cluster is a backport, and the backport bot fails silently on conflict and imports whole files for files absent on the target branch. Security fixes inherit that latency. +- **Nothing can be tested or supported at package granularity,** because nothing *is* a package at release granularity. #43 wants catalog repositories with their own CI against a matrix of supported Cozystack releases; there is no version to put in that matrix except the whole platform's. +- **The word "version" has no defined meaning.** Nobody has had to decide what makes a chart change breaking, because no chart version has ever been read by anything. + +### Why this is a release-engineering proposal, not a marketplace proposal + +Per-package versioning has come up before in an ecosystem context — publishing, discovery, letting third parties ship apps — and in that context it was reasonably set aside. This proposal deliberately does not argue from ecosystem ambition, a roadmap item, or what other platforms have. It argues from four things that hurt today, in a repository with no external catalogs at all: + +1. **A fix cannot reach a released cluster without a platform release.** The only route today is a backport, and the backport automation fails silently on conflict and imports whole files when the target branch lacks the file being modified. Security fixes inherit that latency and that failure mode. +2. **Every upgrade is a full-platform upgrade.** A month whose substance is three bug fixes still moves 160 chart versions, re-reconciles the tree, and asks operators for a maintenance window sized to the whole platform rather than to the change. +3. **Breaking changes are structurally discouraged.** With one version, a breaking change in one app is a platform-level event. The rational response is to defer it, and deferred schema debt is why [#6](https://github.com/cozystack/community/pull/6) exists at all — the shapes it wants to fix (`users` as a map, `postgresql.parameters.max_connections`) have been wrong for a long time and stayed wrong because there was no way to charge the cost to one app. +4. **Release readiness is all-or-nothing.** `v1.6.0` needed four release candidates and a NO-GO on rc.1 for upgrade-only blockers in two components. With a single version, two blocked components block everything; with a manifest, they hold at their previous versions and the train ships. + +None of those four require a marketplace, a community index, or a single external contributor. They are internal costs paid every release cycle by the people running the release. The relevant observation is that **the machinery which fixes them is the same machinery an ecosystem needs** — a versioned pool, a manifest, and a package manager that understands both. That coincidence is worth stating precisely, because it determines build order: the internal requirement demands a lockfile and per-package versions and can be satisfied with no discovery surface whatsoever, while a discovery index is optional sugar on top of it. If the ecosystem work never happens, everything in this proposal is still worth building. If it does happen, it lands on a foundation that already exists rather than requiring one. + +## Goals + +- A Cozystack release is a **manifest** naming a core version and an exact version for every package and app it ships, and that manifest is a published artifact users and tooling can read. +- Upgrading a release upgrades only the components whose versions changed. Unchanged components are not re-reconciled. +- A package or app can ship a breaking change on its own major version without a platform major. +- A security fix in one package can be released and consumed without a platform release. +- "Breaking" has a written, CI-enforceable definition for a Cozystack package. +- External catalogs (#43 tier 1/2) and community repositories (#18) are the same mechanism as first-party packages, differing only in origin and support level. +- Release cadence is predictable one to two months ahead, with a published support window. + +### Non-goals + +- **Arbitrary user-composed version mixing.** A release pins a tested set. Users select a release, not a basket of versions. Overrides exist (see [Holds and overrides](#holds-and-overrides)) but are explicitly unsupported territory, exactly as a distribution treats a pinned package from a foreign repository. +- Changing how Talos, the installer, or cluster bootstrap work. +- Designing the conversion mechanism for app APIs — that is #6. +- Deciding the tier of any specific application — that is #43. +- A hosted registry service. Artifacts live in ordinary OCI registries, as they do today. + +## Design + +### 1. Four tiers, one package model + +| Tier | Contents | Versioning | Cadence | +|---|---|---|---| +| **Core** | `cozystack-api`, `cozystack-controller`, `cozystack-operator`, the `cozystack.io` CRDs, and the delivery contract (`PackageSource`, `Package`, `ApplicationDefinition`, `ApplicationGroupDefinition`) | SemVer | On its own, as needed | +| **Packages** | Platform components — operators, CSI/CNI, monitoring, ingress; today's `packages/system` | SemVer, Cozystack's own, not upstream's | Continuous, released from `main` | +| **Apps** | User-facing managed applications; today's `packages/apps` (and `packages/extra` after #39) | SemVer | Continuous, released from `main` | +| **Distribution** | The thing users install and name | CalVer `YYYY.MM` | Monthly | + +Mechanically, packages and apps are the **same object** — both are `PackageSource` + `Package` + charts, and both should stay that way. The distinction is a `section` on the package (Debian's `Section:` field), used for catalog presentation, tiering, and support policy, not for delivery. This mirrors #39's central argument one level up: a directory is not a mechanism, and behaviour that differs should be a declared field. + +Core is the one genuine exception, and it must be defined by **contract, not by binary**. If core's version tracks the Go code, a controller bug fix and a CRD field removal both read as "core changed" and the number communicates nothing. Core's semver describes the compatibility of the CRD groups, the `PackageSource`/`Package`/`ApplicationDefinition` semantics, and the operator's artifact-resolution behaviour. The binaries are implementation. + +### 2. The distribution mapping + +Naming the analogy precisely is useful, because it makes each missing piece obvious and tells reviewers which prior art to argue from. + +| Debian / Ubuntu | Cozystack today | Under this proposal | +|---|---|---| +| Release (`bookworm`, `24.04`) | a git tag → one OCI digest | CalVer release = a manifest pinning exact versions | +| `base-files` / base system | core | Core, semver, own cadence | +| Package pool (`pool/main`) | one OCI artifact holding all of `packages/` | one artifact per package per version | +| `Release` / `Packages` index | *does not exist* | the release manifest **is** the index | +| `main` / `universe` / `multiverse` | in-tree only | #43's tiers 0 / 1 / 2 | +| `control` file | scattered: `Chart.yaml`, the `-rd` ApplicationDefinition, the `PackageSource` entry | ApplicationDefinition, extended by #39 and this proposal | +| Metapackage (`ubuntu-desktop`) | `templates/bundles/{system,iaas,paas,naas}.yaml` | unchanged, now version-pinned | +| `dpkg --get-selections` | `Package` CRs with `helm.sh/resource-policy: keep` | **already correct, no change** | +| `apt` | `cozypkg add/del/list/dot` | + versions, repositories, search, upgrade, hold | +| Maintainer scripts (`preinst`) | 53 global numbered shell scripts | per-package pre-upgrade hooks | +| `apt-mark hold` | nothing | version override on the `Package` CR | +| PPA / third-party repo | `external-apps-example` (legacy path) | #18 taps, #43 catalogs | + +Two of these already work and are simply unnamed. Bundles (`packages/core/platform/templates/bundles/*.yaml`, rendered through the four helpers in `_helpers.tpl:1,27,33,52`) are metapackages complete with opt-in and opt-out (`values.yaml:37-59` — `bundles..enabled`, `enabledPackages`, `disabledPackages`, plus per-component `Package.spec.components[].enabled`). And because bundle-emitted `Package` CRs carry `helm.sh/resource-policy: keep` (`_helpers.tpl:17`), removing a package from values does not uninstall it — the installed set is already a separate thing from the shipped set, which is precisely the selections semantics a package manager needs. + +### 3. The release manifest + +The manifest is the central new artifact. It is what a release *is*. + +```yaml +apiVersion: cozystack.io/v1alpha1 +kind: Release # exact kind TBD — see Open questions +metadata: + name: "2026.08" +spec: + channel: stable # stable | rc | nightly + core: 1.7.2 + supersedes: "2026.07" + supportedUpgradeFrom: ["2026.07", "2026.06"] + migrationFloor: 54 # see Migrations + packages: + - name: cozystack.linstor + section: system + version: 2.4.0 + digest: sha256:… + - name: cozystack.postgres-application + section: apps + version: 3.2.1 + digest: sha256:… + requiresCore: ">=1.7.0 <2.0.0" + # … + repositories: # additional origins shipped enabled-by-default + - name: cozystack-catalog + url: oci://ghcr.io/cozystack/catalog + version: 2026.08 +``` + +Properties that matter: + +- **The manifest is the index.** #18's meta-index and #12's community index are the same object shape for a different origin. There is one reader. +- **It is a diffable artifact.** `2026.09` minus `2026.08` is the release notes, mechanically. This replaces changelog generation that today has to summarise "everything that landed". +- **It is the mirror list.** For #21's air-gapped bundle, the manifest enumerates exactly what must be pulled — currently that enumeration only exists inside `hack/lib/image-refs.sh` and its four consumers. +- **`requiresCore` is #43's "declared minimum platform version",** which #43 already assumes is checked at `PackageSource` reconciliation. This is where it comes from. + +### 4. The versioned pool — two options and the experiment that decides + +**Option A — versioned paths in one artifact.** Keep publishing one OCI artifact per release, but version the paths inside it: `apps/postgres/3.2.1/…`. A partial upgrade means the new manifest points unchanged packages at unchanged paths. If the content copied by the `ArtifactGenerator` is byte-identical, the resulting `ExternalArtifact` is identical, helm-controller sees no new revision, and the HelmRelease is not upgraded. Nearly no API change; a build-system change and a path convention. + +**Option B — one artifact per package per version.** `PackageSourceRef` (or `Component`) gains a `version`, and the reconciler emits one source alias per distinct `(repository, version)`. This is the real repository model and the one external catalogs need anyway. + +Everything hinges on one unknown: **is `ExternalArtifact` revision content-derived or source-derived?** `ArtifactGenerator.spec.artifacts[].revision` must be an `@` pointer, which suggests the revision tracks the *source*, not the copied content. If so, every package's artifact revision changes whenever the bundle digest moves, every HelmRelease reconciles, and Option A delivers nothing. + +**The experiment, to be run before this proposal is accepted:** publish two `cozystack-packages` artifacts differing in exactly one package; point the platform at the second; observe whether an untouched component's `ExternalArtifact` `status.artifact.digest` and `.revision` change, and whether its HelmRelease reconciles. This is an afternoon on a dev cluster and it decides whether the proposal is a build change or an API change. The result should be recorded in this document before the design is locked. + +Encouragement for Option B: `ArtifactGenerator.spec.sources[]` is **already a list with aliases** upstream, and the copy operations already address `@alias/path/**`. Cozystack always writes exactly one entry (`packagesource_reconciler.go:268-275`). The change is confined to `reconcileArtifactGenerators`. The cost is object count — from 2 `OCIRepository` objects to roughly 100, each with a 5-minute poll interval, plus source-watcher's `emptyDir` unpack (`internal/fluxinstall/manifests/fluxcd.yaml:8287-8288`) rebuilding on every pod restart. That must be load-tested, not assumed. + +A third structural coupling to resolve under either option: `package_reconciler.go:131` looks up the `PackageSource` **by the same name as the `Package`**. That 1:1 naming is fine if version lives as a field on those objects, and blocks anything that would need parallel objects per version. + +### 5. What MAJOR means + +SemVer without a written rule is noise, and this is the item most likely to quietly kill the initiative. The definition proposed here anchors on #6, which is what makes a breaking change survivable rather than merely announced: + +| Bump | Definition | Consequence | +|---|---|---| +| **PATCH** | No change to the values schema, no change to any resource identity | Upgrade in place, no migration, no notice | +| **MINOR** | Additive schema only (new optional keys with defaults); no removed or retyped keys; no immutable-field or selector change | Upgrade in place; may ship a per-package migration; release-note entry | +| **MAJOR** | Any of: a removed or retyped values key; a change of storage version under #6; a workload rename or immutable-field change requiring adoption or recreation; a change to a capability the package provides to others (#39) | Requires a conversion path (#6) or a documented manual action; may not be crossed by a partial upgrade without an explicit gate | + +Two CI gates make this real, and both must ship in the same phase as the version numbers: + +1. **Content changed implies version bumped.** A package whose rendered output differs from the previous release without a version bump fails the build. Without this, an unchanged chart version over changed content becomes an OCI/Flux cache-poisoning bug. +2. **Schema-diff classifies the bump.** Compare the generated OpenAPI schema against the previous version and assert the bump is at least as large as the diff implies. Removal or retyping demands MAJOR. + +The second gate is where #6 and this proposal meet: #6 gives an app the ability to *survive* a storage-form change, and this proposal gives it the number that *advertises* one. A MAJOR without a `to`/`from` pair is a manual-action release; with one, it is transparent. + +### 6. Migrations: per-package, generalising a pattern that already exists + +Today all 53 migrations live in `packages/core/platform/images/migrations/migrations/` as bare numbered shell scripts, are selected by `run-migrations.sh:39` walking `seq $CURRENT_VERSION $((TARGET_VERSION - 1))`, run as **cluster-admin** in one Job, and execute as a `pre-upgrade` hook on the `cozystack-platform` Helm release — which means *all* of them run before *any* component HelmRelease upgrades. The only state is `data.version` in the `cozy-system/cozystack-version` ConfigMap; there is no per-migration record, so every script must be idempotent, and that is convention rather than enforcement. + +Under partial upgrades a single global counter has no defined meaning. But the redesign is less open-ended than it looks, for two reasons. + +**First, the ordering requirements are already per-package.** Examining what the recent migrations actually need: + +| Migration | Must run before | Owning package | +|---|---|---| +| 45, 46, 47 | the `kubernetes` chart re-renders (else Helm prunes live `KubeadmConfigTemplate`/`MachineDeployment` objects, or the chart `fail()`s on `v1.30`) | `packages/apps/kubernetes` | +| 48 | ClickHouse keeper GC hook selects PVCs | `packages/apps/clickhouse` | +| 51 | monitoring PVC selectors apply | `packages/system/monitoring` | +| 52 | the linstor release upgrades (immutable `spec.selector`) | `packages/system/linstor` | +| 43, 53 | the seaweedfs release re-renders (else the CNPG `Cluster` and all tenant S3 metadata are pruned) | `packages/extra/seaweedfs` | +| 49 | tenant namespace policies apply | `packages/apps/tenant` | +| 44 | (deferred by design — waits on runtime drain) | `packages/system/flux-shard-operator` | +| 50 | the etcd-operator HelmRelease reconciles | **cross-package**: etcd-operator, etcd-operator-crds, backupstrategy-controller, seaweedfs, extra/etcd | + +Every one of those is "before *my own* package upgrades" — which a per-package `pre-upgrade` hook provides natively, with a stronger guarantee than today's implicit "the platform hook happens to run first". Nine of the last ten migrations attribute cleanly to exactly one package, and the commit messages already name the owner (`fix(monitoring):`, `fix(linstor-scheduler):`, `fix(clickhouse):`, `refactor(seaweedfs):`). Migration 50 is genuinely cross-cutting and stays global. + +**Second, the per-package pattern is already implemented four times in-tree:** + +- `packages/system/seaweedfs/templates/hook.yaml` + `templates/version.yaml` — a complete miniature of the platform mechanism, with its own `seaweedfs-deployed-version` ConfigMap and a `lookup`-gated `pre-upgrade` Job +- `packages/system/etcd-operator/templates/pre-upgrade-selector-fix.yaml` — conditional on the live object's state rather than a counter +- `packages/system/dashboard/templates/adopt-configmap-hook.yaml` +- `packages/apps/vm-disk/templates/pvc-resize-hook.yaml`, `packages/apps/vm-instance/templates/vm-update-hook.yaml` + +The design is therefore: **promote that pattern into `packages/library/cozy-lib`**, which is already injected as a Helm subchart per component by the artifact generator (`packagesource_reconciler.go:212-219`), so every package gets a migration framework for free. Each package owns a counter in its own release; the global lane survives, owned by core, for the cross-cutting class (migration 50) and the platform-config migrations (7, 12, 21, 25, 31, 32, 42). `migrationFloor` in the release manifest records the global counter, so the two coexist. + +Three details to carry across in the generalisation: + +- The seaweedfs guard compares versions as **strings** (`ge $deployedVersion "3"`), which breaks at 10. Fix it in the library, not in each copy. +- Per-package hooks run in the package's own release with the package's own RBAC. This retires the cluster-admin blast radius for everything except the global lane — a hard requirement once external catalogs ship migrations (see [Security](#security)). +- Per-package counters retire the cross-branch hazard documented at `docs/release.md:357-367`, where backporting a migration into a maintenance branch burns that slot on `main` forever. + +### 7. `cozypkg` is the package manager, and it is 80% unwritten + +`cmd/cozypkg` is 1811 lines across five files, three commits, and reads or writes exactly two CRDs. It has `add`, `del`, `list`, `dot`. It has no concept of a version, a repository, an index, or a search. `cozypkg list` is effectively `kubectl get packagesources`. It is built for six platforms (`Makefile:96`), uploaded to every release (`hack/upload-assets.sh:15-16`), and mentioned in **zero lines of documentation** anywhere in the repository. + +If Cozystack is a distribution, this is the user-facing surface of the entire proposal. The command surface needed is close to what #12 already specified, plus versions: + +| Command | Status | Purpose | +|---|---|---| +| `cozypkg add` / `del` / `list` / `dot` | exists | unchanged semantics | +| `cozypkg list --available` | extend | packages from the release manifest, not only `PackageSource` objects already on the cluster | +| `cozypkg search ` | #12 | across manifest and tapped repositories | +| `cozypkg show ` | new | installed version, available version, section, tier, provenance | +| `cozypkg upgrade [pkg]` | new | move to the version in the current manifest | +| `cozypkg hold` / `unhold` | new | pin a package across releases; see below | +| `cozypkg tap` / `untap` | #12, #18 | register an external repository, optionally `--secret` (#23) | +| `cozypkg init` / `push` / `validate` | #12, #23 | authoring workflow | +| `-o yaml/json`, `--yes` | new | it is currently interactive-only, which blocks all automation | + +Two existing defects to fix while touching it: `add -f` on an existing Package fails the create and the error is silently swallowed (`add.go:116-121`), falling through to the interactive path — it is create-only with no apply semantics; and the package dependency graph has **no cycle detection** in `package_reconciler.go`, so a cycle deadlocks both packages in `DependenciesNotReady` forever with no diagnostic. Acceptable with 98 curated packages; not acceptable with open repositories. + +#### Holds and overrides + +`Package.spec` gains an optional `version` that overrides the manifest. Setting it is `apt-mark hold`: supported as a mechanism, unsupported as a configuration. `cozypkg list` and the dashboard mark held packages, and the cluster's version vector reports them (see [Diagnostics](#diagnostics)). This is the escape hatch that makes a strict manifest tolerable in the field without turning the support matrix into a combinatorial space. + +### 8. Repository-as-unit versus package-as-unit + +This is the one place where this proposal reaches a different conclusion from an existing one, so it is set out in full rather than elided. + +**The history matters, because it is a considered position and not an oversight.** [#12](https://github.com/cozystack/community/pull/12) (2026-05-26) was package-centric: publish, index, and install individual packages. [#18](https://github.com/cozystack/community/pull/18) (2026-06-23) proposed the repository as the unit instead, listing the package-centric model under Alternatives considered and rejecting it on the grounds that a thematic repository carries a "tested together" guarantee that a loose package catalog does not. #12 was then closed on 2026-07-16. For the problem #18 addresses — how a community publishes coherent, mutually-tested sets of applications that Cozystack maintainers have not reviewed — **that reasoning is correct, and this proposal adopts #18's model unchanged for that case.** A third-party repository is authored and tested as a set, its author owes Cozystack no compatibility guarantee, and a repository-level tag costs nothing because the OCI artifact already exists. + +The different conclusion here comes from a requirement neither #12 nor #18 was scoped to weigh: **the platform's own release engineering.** #18's "tested together" argument is exactly the argument for a manifest — and a manifest that can only name whole repositories cannot express "these three packages moved and the other 155 did not". Shipping `2026.09` with the same core and three bumped apps is the entire value of the partial-upgrade goal, and it is unreachable if the finest addressable unit is the repository. So the disagreement is narrow and does not touch #18's thesis: it is about whether the *first-party archive* is one repository or many packages, not about how community repositories should work. + +Both are true at different tiers, which is also how Debian works — the archive is versioned per package and resolved by a release; a third-party PPA is versioned as a unit and you take what it gives you. + +Two practical notes for whoever reconciles these. First, adopting per-package versioning for the first-party archive costs #18 nothing: its meta-index, tap flow, and repository-level tags are unaffected, and the manifest reader this proposal needs is the same `TapIndex` cache [#23](https://github.com/cozystack/community/pull/23) already specifies. Second, #12's concrete surface — `tap` / `untap` / `init` / `push` / `search`, `community.`-prefixed source names, metadata-only index entries with an optional expected signing identity — is reusable as written; the package-level axis is being revived here on new grounds, not the specific ergonomics being re-litigated. + +| Origin | Versioned unit | Rationale | +|---|---|---| +| Platform (tier 0, in-tree) | per package | partial upgrades; support matrix; migrations | +| Curated catalog (#43 tier 1) | per package, published in the catalog's own manifest | same guarantees, different maintainers | +| External catalog (#43 tier 2) / community tap (#18) | the repository (its OCI tag) | tested-together guarantee; no compatibility obligation | + +### 9. ApplicationDefinition as the control file + +Three proposals are independently adding fields to the same object, and it is worth naming what it is becoming. `ApplicationDefinition` already carries kind, plural/singular, the OpenAPI schema, the release prefix and `chartRef`, dashboard metadata, and secret/ingress projections. #39 adds visibility, cardinality, protection, and capability provides/consumes. #6 adds `versions[]` with a storage version and conversion templates. cozystack#3448 adds a group selector. This proposal adds the package version, its section, its dependency constraints, and `requiresCore`. + +That is a Debian `control` file. Recognising it has one practical consequence worth acting on now: the object is accumulating fields from four directions at `v1alpha1`, external catalogs are about to depend on its shape across repository boundaries (#43 flags exactly this), and nobody owns its coherence. **This proposal recommends a single consolidating pass on `ApplicationDefinition` — one PR, one shape, all four proposals' fields reviewed together — before any of them ships its fields independently.** Otherwise the cross-repo contract that #43 needs stabilised gets four uncoordinated `v1alpha1` extensions first. + +### 10. Bundles stay, and gain versions + +No change to the bundle mechanism is proposed. `templates/bundles/{system,iaas,paas,naas}.yaml` keep emitting `Package` CRs through the `_helpers.tpl` helpers, `enabledPackages` / `disabledPackages` keep working, and `helm.sh/resource-policy: keep` keeps meaning "removing this from values does not uninstall it". What changes is that the versions the bundle installs come from the release manifest rather than from whatever happens to be in the tree. + +Worth noting for reviewers that the bundle layer already carries the correct semantics for a distribution and is under-documented: it supports whole-bundle toggles, opt-in of optional packages (`nfs-driver`, `telepresence`, `external-dns`, `kuberture`, `external-secrets-operator`, `linstor-gui`, `hetzner-robotlb`, `bootbox`, `vm-default-images`, `hami`, `gpu-operator`), opt-out of anything, and per-component disable. + +## Release cadence and branch model + +The cadence change is separable from everything above and should ship first, because it delivers most of the operational benefit at almost no implementation cost and produces two or three cycles of evidence before the versioning work bets on it. + +A four-week train: + +| Week | Where work lands | What happens | +|---|---|---| +| 1–2 | `main` | Normal development. Features merge to `main` as today | +| 3 | `release-YYYY.MM` branch cut | Stabilisation. Bug fixes; plus features that must ship this month because they were promised or because they complete something already in the train. No speculative features | +| 4 | same branch, RC tags only | No new features. RCs are cut and tested on dev clusters and with bleeding-edge customers | +| end | tag | Stable release, or the month is skipped | + +```mermaid +gantt + dateFormat YYYY-MM-DD + axisFormat %d %b + section main + development (continuous) :active, m1, 2026-08-01, 28d + section release-2026.08 + cut + stabilisation :s1, 2026-08-15, 7d + RC testing (no new features) :s2, after s1, 7d + stable tag :milestone, after s2, 0d +``` + +Three rules make it hold: + +1. **A train that is not ready skips the month; it never extends.** A calendar name makes slipping impossible to hide, and v1.6.0 needed four release candidates with a NO-GO on rc.1 for upgrade-only blockers. `2026.09` not existing is a clean, legible outcome. `2026.08` shipping on 12 September is not. +2. **Packages do not get release branches.** Only the train does. Packages are released from `main` and referenced by version in the manifest. Otherwise the branch topology multiplies by the number of packages. +3. **The support window is published before the cadence changes.** Twelve releases a year cannot each be supported. The proposal recommends stating it explicitly — for example, the current release plus the two preceding, with security fixes only for the older two — and making `supportedUpgradeFrom` in the manifest the machine-readable form of that promise. + +### Core releases between trains + +Because core is not low-churn (see [Context](#core-is-not-low-churn)), core will be ready to release more often than monthly, and the design must say what happens then. Two options, and the proposal recommends the first: + +- **Recommended — core ships only at train boundaries, but may bump more than one minor per train.** The manifest names one core version per release. Out-of-band core releases exist as artifacts (for testing and for catalog CI) but are not installed by a stable release. Simple to support; the cost is that a core fix waits up to four weeks. +- Core ships out of band, and a cluster's effective core version may be newer than its release stamp. Faster, but it makes the release stamp a partial description of the cluster and complicates every support conversation. + +Either way, security fixes need an exception path: a patch release of the current train (`2026.08.1`) that changes only the affected component's version in the manifest. That is the mechanism the whole proposal buys, and it should be exercised deliberately at least once per quarter rather than discovered during an incident. + +## User-facing changes + +- **Version strings change.** A cluster is "Cozystack 2026.08" running "postgres 3.2.1 (PostgreSQL 16.2)". Three numbers where there was one. The dashboard, `cozypkg`, and the docs must render this coherently, and app cards should show the package version and the upstream `appVersion` distinctly — users already see `appVersion` today and will otherwise conflate the two. +- **`cozypkg` becomes a real CLI** with versions, search, upgrade, and repositories — and gains documentation, which it has never had. +- **Upgrades get smaller.** A monthly release that touched three packages upgrades three packages. +- **Release notes become mechanical** — a manifest diff. +- **A new artifact exists:** the release manifest, published per release and per channel. +- No change for tenants. Tenant-facing `Application` resources, their kinds, and their API groups are untouched by this proposal; API-surface evolution is #6's and #3448's territory. + +## Upgrade and rollback compatibility + +- **The first release under this model must be a no-op in behaviour.** Ship the manifest with every package version equal to the current platform version, resolve everything the way it resolves today, and change nothing else. This is the single most important sequencing rule in the proposal: it makes the machinery testable before it carries any semantics. +- **Existing clusters** keep working — all new fields are additive with today's behaviour as the default, exactly as #39 and #3448 do. +- **Rollback of a release** becomes rollback of a manifest, which is strictly better than today: currently rolling back means moving the whole platform digest. +- **Rollback across a package MAJOR** is not automatic and must be documented per package. Where #6's conversion templates exist and round-trip, it is; where a MAJOR involved a workload rename or an immutable-field change, it is not. +- **Downgrade of the global migration counter** keeps its current semantics; per-package counters follow the same rule — a package's migration does not run backwards. + +## Security + +- **Per-package migration hooks retire a large blast radius.** The current runner is one Job with `cluster-admin` (`migration-hook.yaml:62-86`) executing arbitrary shell from the platform image. Per-package hooks run inside the package's release with that package's RBAC. +- **External packages must never reach the global migration lane.** This is a hard boundary, not a guideline. The global lane is core-owned; a tapped repository ships per-package hooks only. +- **Installing a repository stays a cluster-admin action,** as #43 and #18 both state: a catalog causes the platform to render and apply arbitrary Helm charts. +- **A versioned pool makes signing worth standardising.** #43 lists artifact signing as an open question; with per-package artifacts the manifest is the natural place to carry expected signing identity per package, and #12 already anticipated an "expected signing identity" field in its index entries. +- **Private repositories** are handled by #23's `secretRef` threading; nothing here changes it. +- **No new tenant-facing surface.** Everything in this proposal is admin-facing or build-time. + +## Failure and edge cases + +- **A package version in the manifest does not exist in the registry** → the source reports not-ready and the release does not partially install. Validated at release-build time, not at install time. +- **`requiresCore` unsatisfied** → `PackageSource` reports not-ready with an explicit message rather than half-installing. This is #43's stated behaviour for catalogs, applied uniformly. +- **A held package blocks an upgrade** whose other components require a newer version of it → the upgrade refuses with the conflict named, rather than proceeding into an untested combination. +- **Content changed without a version bump** → build fails (CI gate 1). Without this gate the failure mode is a silently stale artifact served from cache. +- **A package graph cycle across repositories** → currently a silent permanent deadlock; cycle detection with a surfaced condition is required work in this proposal. +- **Two repositories ship the same package name** → `community.`-prefixed naming per #12; first-write-wins within an origin, as today. +- **A partial upgrade that skips a package with a pending migration** → the package's own counter is unchanged, so its migration runs whenever that package is next upgraded. This is the property the global counter cannot express. +- **Migration hook fails** → today this fails the platform HelmRelease and blocks the entire upgrade; per-package, it fails that package's release and the rest of the train proceeds. This is an improvement in blast radius and a change in behaviour that must be called out in release notes. + +## Testing + +- **Conformance, in-tree and gating:** a synthetic package published at two versions; assert that upgrading the manifest from v1 to v2 reconciles exactly that package and leaves an untouched neighbour's HelmRelease revision unchanged. This is the acceptance test for the entire proposal, and it doubles as the [experiment](#4-the-versioned-pool--two-options-and-the-experiment-that-decides) that chooses Option A or B. +- **CI gates:** content-changed-implies-bump, and schema-diff-classifies-bump, both run per PR over the package tree. +- **Migration framework:** unit tests on the `cozy-lib` version helper including the string-vs-integer comparison at 10; per-package hook tests reusing the existing helm-unittest harness. +- **Upgrade e2e:** a two-release chain (`2026.08` → `2026.09`) with a partial manifest diff, asserting untouched workloads are not restarted. The existing seed → upgrade → verify lane is the right host for this. +- **Manifest validation:** every package in a manifest resolves; `requiresCore` is satisfiable; `supportedUpgradeFrom` chains are acyclic and reachable. +- **Load:** if Option B is chosen, source-controller and source-watcher behaviour with ~100 `OCIRepository` objects, including source-watcher pod restart with a cold `emptyDir`. + +## Rollout + +The ordering is chosen so that each phase is independently valuable and independently revertible, and so that the riskiest item (the versioned pool) is preceded by the experiment that de-risks it. + +1. **Cadence only.** Adopt the four-week train, publish the support window, cut `release-YYYY.MM` branches. No code changes. Two or three cycles of evidence before anything else depends on the cadence holding. +2. **Experiment and decision.** Run the `ExternalArtifact` revision experiment; record the result in this document; choose Option A or Option B. +3. **Manifest, inert.** Publish the release manifest with all versions equal to the current platform version. Add `version` to the data model. Nothing resolves differently. Add the two CI gates. +4. **ApplicationDefinition consolidation.** One coordinated pass folding this proposal's fields with #39's, #6's, and #3448's, before external catalogs depend on the shape. +5. **Migration framework.** `cozy-lib` migration helper; port the four existing ad-hoc hooks onto it; new migrations are authored per-package; the global lane is frozen except for cross-cutting cases. +6. **First real partial upgrade.** One release where the manifest moves a small number of low-risk packages (`bucket`, `http-cache` — not `postgres`, not `kubernetes`) and everything else holds. Measure what the tooling misses. +7. **`cozypkg`.** Versions, search, show, upgrade, hold, non-interactive output, and documentation. +8. **Repositories.** #43's catalogs and #18's taps land on the now-versioned mechanism; the manifest's `repositories` list ships the org catalog enabled-by-default per #43's shim plan. +9. **CalVer switch.** Rename the release stream once the manifest, partial upgrades, and the support window are all proven. Last, not first. + +## Open questions + +- **Option A or Option B** — resolved by the experiment in phase 2. Everything else in the design is stable across the answer; only the implementation site changes. +- **Manifest kind and home.** A CRD applied to the cluster, a plain OCI artifact read by tooling, or both? #23's `TapIndex` cache is the obvious reader either way. +- **How does the CalVer name relate to the existing `v1.x` stream?** Is `2026.08` a rename of the same stream, or does a final `v1.x` release announce the switch? What do existing release branches and backport automation do at the boundary? +- **Support window length.** Three releases? Six? Security-only tail? This must be answered before phase 1, not after. +- **Does core ship between trains?** Recommendation above is no; maintainers should confirm. +- **Version reporting and diagnostics.** With holds and partial upgrades, a cluster's state is a version vector rather than a single string. `cozypkg`, the dashboard, and the diagnostic bundle (`cozyreport` / crust-gather) must all carry it, or support gets harder rather than easier. Who owns that surface? +- **Inter-package compatibility.** Debian works because packages have declared ABIs. Helm charts have none — the real interface is the values schema plus what one chart `lookup`s about another's live state, and several charts do exactly that (the Harbor jobservice storage-class preservation, the SeaweedFS fullname adoption). Should cross-package `lookup` be forbidden outside core, or should packages declare an interface version? +- **Air-gap.** #21's bundle currently mirrors one artifact. With a per-package pool, what does the bundle contain, and does the manifest become the mirror spec? Coordinate with #21 rather than deciding here. +- **The `api/apps/v1alpha1` Go module** is tagged in lockstep with the platform and mirrors app schemas for external consumers. Per-app modules, or an explicit statement that the module tracks core rather than apps? +- **Naming.** "Variant" already means two different things — the installer's `talos|generic|hosted` and `PackageSource.spec.variants[]`. Adding "version", "section", "repository", and "channel" to the same vocabulary needs a glossary, or reviews will go sideways. + +## Alternatives considered + +- **Keep one version, improve the tooling.** Better changelogs and better test-impact analysis reduce the symptoms and leave the structure: every release still moves every chart, a breaking app change still forces a platform decision, and a package fix still cannot ship without a platform release. +- **SemVer for the distribution too, no CalVer.** Rejected because the distribution's version has no honest semantic meaning once components carry their own — a monthly release containing one app major and forty patches is neither major nor minor. A date is truthful. It also removes the perverse incentive to avoid necessary breaking changes because "we are not ready for 2.0". +- **CalVer everywhere, including packages.** Rejected: package consumers need compatibility information from the version, which is exactly what a date does not carry. +- **Repository-as-unit for everything (strict #18).** Adopted unchanged for community repositories, where its "tested together" reasoning holds. Not adopted for the first-party archive, for one reason: a repository-level version cannot express a partial upgrade, which is the proposal's primary goal. See [§8](#8-repository-as-unit-versus-package-as-unit) for why this is a narrow disagreement about the archive rather than a rejection of #18's model. +- **Wait for the marketplace work and take per-package versioning as a side effect of it.** Rejected on sequencing. The internal costs enumerated in [Why this is a release-engineering proposal](#why-this-is-a-release-engineering-proposal-not-a-marketplace-proposal) are paid every cycle now and do not depend on any ecosystem work landing; tying their fix to a discovery surface that is still under discussion delays it for reasons unrelated to it. The dependency runs the other way — the marketplace benefits from a versioned archive, not the reverse. +- **Per-package versions but no manifest — resolve with SemVer ranges at install time.** Rejected firmly. Ranges without a lockfile import npm's resolution problem into a platform with no lockfile and no ability to test the resolved set. The manifest *is* the lockfile. +- **Let users compose versions freely.** Rejected as a supported mode. A distribution decides the set; that is what makes it testable and supportable. Holds exist as an escape hatch and are labelled unsupported. +- **Split the repository first, version second** (i.e. do #43 before this). Not rejected — these are compatible and #43 has the more urgent driver in e2e cost. The note is only that #43's catalogs will need a version axis and will otherwise invent a local one per catalog. +- **Design a new migration framework from first principles.** Rejected in favour of generalising the four per-package hooks already in the tree. The existing pattern is proven, and its ordering guarantees are strictly better than the global counter's. + +## Appendix A — measured facts + +Collected from `main` on 2026-07-27 (post-`v1.6.0`). Line references are to that state. + +| Fact | Value | Source | +|---|---|---| +| Charts on the `0.0.0` build-time placeholder | 160 of 164 (excluding vendored `charts/`) | `packages/**/Chart.yaml` | +| Charts already carrying a hand-maintained version | 4 — `apps/foundationdb` `0.1.0`, `system/cozystack-scheduler` `0.3.0`, `system/kubeovn` `0.38.0`, `tests/cozy-lib-tests` `0.1.0`. Nothing reads them; the inconsistency is itself evidence that the convention is unenforced | same | +| Package directories | 23 apps, 126 system, 8 extra | `packages/` | +| `PackageSource` objects shipped | 98, all referencing `cozystack-packages` | `packages/core/platform/sources/*.yaml` | +| `OCIRepository` objects in play | 2 (`cozystack-platform`, its clone `cozystack-packages`) | `main.go:564-602`, `repository.yaml:19` | +| Version fields in the delivery data model | 0 | `packagesource_types.go`, `package_types.go` | +| Migrations | 53, contiguous, `targetVersion: 54` | `images/migrations/migrations/`, `platform/values.yaml:17` | +| Migrations attributable to one package (last 10) | 9 of 10 | commit history on those files | +| Existing per-package migration hooks | 4 | seaweedfs, etcd-operator, dashboard, vm-disk/vm-instance | +| `cozypkg` size / commits / documentation | 1811 lines / 3 commits / 0 lines of docs | `cmd/cozypkg/`, `git log` | +| Core code churn, 12 months | ~620 commits across `internal/`, `pkg/`, `api/` | `git log --since='12 months ago'` | + +## Appendix B — unrelated defects found while surveying + +Not part of this proposal; recorded because they were found in the same pass and are cheap to fix independently. + +- `packages/core/platform/templates/apps.yaml:3-4` assigns `$bundle` from `.Files.Get "bundles/.yaml"`, a path that does not exist in the chart root, and never reads the result. Vestigial from before the operator rewrite. +- `packages/core/platform/values.yaml:40` still advertises `"distro-full"` as a valid system variant. It was removed around v1.0.1 and is not in the operator's variant table (`main.go:678-686`); setting it produces an empty system bundle and a hard `fail` from the other three bundles. +- The second variant in `sources/capi-provider-{bootstrap-kubeadm,cp-kamaji,infra-kubevirt}.yaml` is byte-identical to `default` and unreachable — the bundles call the `.default` helper for all three. +- `packages/system/opencost/` is referenced by nothing under `packages/core/` or `hack/`. +- Six `PackageSource` objects are in no bundle and reachable only via `cozypkg add`: `monitoring`, `ingress-nginx`, `local-ccm`, `clustersecret-operator`, `cluster-autoscaler-hetzner`, `cluster-autoscaler-azure`. +- `cmd/cozypkg/cmd/add.go:116-121` swallows the create error when `add -f` targets an existing `Package`, silently falling back to the interactive flow, which then reports "already installed" and does nothing. +- `internal/operator/package_reconciler.go` has no cycle detection; a dependency cycle leaves both packages in `DependenciesNotReady` indefinitely with no surfaced diagnostic. +- `packages/system/seaweedfs/templates/version.yaml` compares its deployed version as a string (`ge $deployedVersion "3"`), which will misbehave at version 10. + +--- + + From 026f876429e059d8f65140abfd043e12707e83f2 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Tue, 28 Jul 2026 12:29:27 +0500 Subject: [PATCH 02/13] docs(cozystack-as-a-distribution): resolve the artifact-revision question from source The proposal gated its implementation choice on whether ExternalArtifact revision is content-derived or source-derived, and proposed a cluster experiment to settle it. It is settled from source instead, and the answer enables Option A. The upstream ArtifactGenerator CRD documents spec.artifacts[].revision as optional, defaulting to "the digest of the artifact content" (internal/fluxinstall/manifests/fluxcd.yaml:522-528), and cozystack never sets it -- reconcileArtifactGenerators builds OutputArtifact{Name, Copy} and the string Revision does not appear in packagesource_reconciler.go at all. The earlier reading of the "@" pattern as evidence of source-derivation was wrong: it constrains the field when set. This is the strongest finding in the proposal, because it inverts the causality. Generated revisions being content-derived means helm-controller already skips a package whose content did not change. The reason every package upgrades on every release is the version stamp itself: rewriting Chart.yaml from COZYSTACK_VERSION for 160 charts guarantees no package is ever unchanged. The single global version number is not just a labelling problem, it is the direct mechanical cause of full-platform upgrade churn -- so giving charts stable versions does not merely enable partial upgrades later, it stops manufacturing the churn that makes them impossible. One narrower question replaces the old one: whether the generated tarball is reproducible for identical input content across two source revisions. If source-watcher preserves unpack-time mtimes, Option A needs an upstream determinism fix alongside it. That is checkable on kind with source-controller and source-watcher alone -- no Cozystack cluster and no dev stand. Updates the overview, the problem statement, section 4, rollout phase 2, testing, open questions, and appendix A accordingly. Assisted-By: Claude Signed-off-by: Myasnikov Daniil --- .../cozystack-as-a-distribution/README.md | 30 ++++++++++++------- 1 file changed, 19 insertions(+), 11 deletions(-) diff --git a/design-proposals/cozystack-as-a-distribution/README.md b/design-proposals/cozystack-as-a-distribution/README.md index 71875dc..502bf8e 100644 --- a/design-proposals/cozystack-as-a-distribution/README.md +++ b/design-proposals/cozystack-as-a-distribution/README.md @@ -11,7 +11,7 @@ Cozystack has exactly one version number, and everything wears it. 160 of the 16 This proposal replaces that single number with four, arranged the way an operating-system distribution arranges its own: a **core** on semantic versioning (the apiserver, controller, operator, CRDs, and the delivery contract itself), **packages** and **apps** each carrying their own semantic version, and a **distribution** — the thing users install and talk about — on calendar versioning, released monthly. A release stops being "the state of a git tree at a tag" and becomes what a distribution release has always been: a manifest that pins an exact set of component versions, tested together, shipped together, supported together. -The change is smaller than it sounds, because the platform is already most of the way there and nobody has been calling the pieces by their distribution names. `PackageSource` and `Package` are a package database. Bundles are metapackages. `helm.sh/resource-policy: keep` on the `Package` CRs means the installed set already diverges from the shipped set — that is `dpkg --get-selections`, implemented by accident. `cozypkg` is `apt` with the version, repository, and index code removed. The per-package pre-upgrade migration hook exists and is used by four packages already. What is genuinely missing is narrow: a versioned pool, a version field on the source reference, and an index. This proposal specifies those three and then connects the in-flight proposals that each assume some part of them. +The change is smaller than it sounds, because the platform is already most of the way there and nobody has been calling the pieces by their distribution names. `PackageSource` and `Package` are a package database. Bundles are metapackages. `helm.sh/resource-policy: keep` on the `Package` CRs means the installed set already diverges from the shipped set — that is `dpkg --get-selections`, implemented by accident. `cozypkg` is `apt` with the version, repository, and index code removed. The per-package pre-upgrade migration hook exists and is used by four packages already. And the delivery layer already does the hardest part: generated artifact revisions are content-derived, so helm-controller already skips a package whose content did not change — the per-release rewrite of every `Chart.yaml` version is what guarantees no package is ever unchanged. What is genuinely missing is narrow: a versioned pool, an index, and eventually a version field on the source reference. This proposal specifies those and then connects the in-flight proposals that each assume some part of them. The case is made from platform release engineering rather than from ecosystem ambition — the costs it addresses are paid every release cycle today, in a repository with no external catalogs at all. That framing is deliberate and is set out in [Why this is a release-engineering proposal](#why-this-is-a-release-engineering-proposal-not-a-marketplace-proposal), because it determines build order: the internal requirement can be satisfied with no discovery surface whatsoever, while a discovery surface cannot be satisfied without it. @@ -85,7 +85,7 @@ One counter-fact to handle: `api/apps/v1alpha1/` is a **separate Go module** (`a ### The problem -- **Every release moves every chart.** Because chart versions are stamped from `COZYSTACK_VERSION`, a release changes the version of 160 charts whether or not their content changed. Users upgrading for a Postgres fix take the whole platform with it. Upgrade windows, blast radius, and rollback all scale with the size of the tree rather than the size of the change. +- **Every release moves every chart, and the version stamp is why.** Because chart versions are rewritten from `COZYSTACK_VERSION` before the pool is pushed, a release changes the version of 160 charts whether or not their content changed — and since generated artifact revisions are content-derived (see [§4](#4-the-versioned-pool)), that stamp is what forces every HelmRelease to upgrade. The delivery machinery would already skip unchanged packages; the single version number is what prevents any package from ever being unchanged. Users upgrading for a Postgres fix take the whole platform with them, and upgrade windows, blast radius, and rollback all scale with the size of the tree rather than the size of the change. - **Breaking changes are held hostage.** A breaking change in one app forces either a platform major (which frightens users away from an upgrade that is mostly bug fixes) or an indefinite delay. There is no way to say "MongoDB 4.0 is breaking, the platform is not". - **A package fix cannot ship without a platform release.** Today the only route to a released cluster is a backport, and the backport bot fails silently on conflict and imports whole files for files absent on the target branch. Security fixes inherit that latency. - **Nothing can be tested or supported at package granularity,** because nothing *is* a package at release granularity. #43 wants catalog repositories with their own CI against a matrix of supported Cozystack releases; there is no version to put in that matrix except the whole platform's. @@ -195,17 +195,23 @@ Properties that matter: - **It is the mirror list.** For #21's air-gapped bundle, the manifest enumerates exactly what must be pulled — currently that enumeration only exists inside `hack/lib/image-refs.sh` and its four consumers. - **`requiresCore` is #43's "declared minimum platform version",** which #43 already assumes is checked at `PackageSource` reconciliation. This is where it comes from. -### 4. The versioned pool — two options and the experiment that decides +### 4. The versioned pool -**Option A — versioned paths in one artifact.** Keep publishing one OCI artifact per release, but version the paths inside it: `apps/postgres/3.2.1/…`. A partial upgrade means the new manifest points unchanged packages at unchanged paths. If the content copied by the `ArtifactGenerator` is byte-identical, the resulting `ExternalArtifact` is identical, helm-controller sees no new revision, and the HelmRelease is not upgraded. Nearly no API change; a build-system change and a path convention. +**Option A — versioned paths in one artifact.** Keep publishing one OCI artifact per release, but version the paths inside it: `apps/postgres/3.2.1/…`. A partial upgrade means the new manifest points unchanged packages at unchanged paths. If the content copied by the `ArtifactGenerator` is byte-identical, the generated artifact is identical, helm-controller sees no new revision, and the HelmRelease is not upgraded. No API change; a build-system change and a path convention. -**Option B — one artifact per package per version.** `PackageSourceRef` (or `Component`) gains a `version`, and the reconciler emits one source alias per distinct `(repository, version)`. This is the real repository model and the one external catalogs need anyway. +**Option B — one artifact per package per version.** `PackageSourceRef` (or `Component`) gains a `version`, and the reconciler emits one source alias per distinct `(repository, version)`. This is the true repository model, and it is what external catalogs need regardless. -Everything hinges on one unknown: **is `ExternalArtifact` revision content-derived or source-derived?** `ArtifactGenerator.spec.artifacts[].revision` must be an `@` pointer, which suggests the revision tracks the *source*, not the copied content. If so, every package's artifact revision changes whenever the bundle digest moves, every HelmRelease reconciles, and Option A delivers nothing. +An earlier draft of this proposal treated the choice as gated on an unknown — whether `ExternalArtifact` revision is content-derived or source-derived — and proposed a cluster experiment to settle it. **That question is settled from source, and the answer enables Option A.** The upstream `ArtifactGenerator` CRD documents `spec.artifacts[].revision` as follows (`internal/fluxinstall/manifests/fluxcd.yaml:522-528`): -**The experiment, to be run before this proposal is accepted:** publish two `cozystack-packages` artifacts differing in exactly one package; point the platform at the second; observe whether an untouched component's `ExternalArtifact` `status.artifact.digest` and `.revision` change, and whether its HelmRelease reconciles. This is an afternoon on a dev cluster and it decides whether the proposal is a build change or an API change. The result should be recorded in this document before the design is locked. +> Revision is the revision of the generated artifact. If specified, it must point to an existing source alias in the format `"@"`. **If not specified, the revision is automatically set to the digest of the artifact content.** -Encouragement for Option B: `ArtifactGenerator.spec.sources[]` is **already a list with aliases** upstream, and the copy operations already address `@alias/path/**`. Cozystack always writes exactly one entry (`packagesource_reconciler.go:268-275`). The change is confined to `reconcileArtifactGenerators`. The cost is object count — from 2 `OCIRepository` objects to roughly 100, each with a 5-minute poll interval, plus source-watcher's `emptyDir` unpack (`internal/fluxinstall/manifests/fluxcd.yaml:8287-8288`) rebuilding on every pod restart. That must be load-tested, not assumed. +The field is optional, and cozystack never sets it: `reconcileArtifactGenerators` constructs `OutputArtifact{Name, Copy}` at `internal/operator/packagesource_reconciler.go:241-244`, and the string `Revision` does not appear anywhere in that file. The `@` pattern constrains the field *when set*; it is not evidence that revisions track the source. So generated artifact revisions are already content-derived, on every cluster running today. + +**This reframes the whole proposal, and it is the most important finding in it.** Content-derived revisions mean helm-controller already ignores a package whose content did not change. The reason every package nevertheless upgrades on every release is the version stamp itself: `packages/core/installer/Makefile` rewrites `Chart.yaml` from `COZYSTACK_VERSION` for all 160 charts before the pool is pushed, so every chart's content changes on every release, so every artifact digest changes, so every HelmRelease upgrades. **The single global version number is not merely a labelling problem — it is the direct mechanical cause of full-platform upgrade churn.** Giving charts real, stable versions does not merely enable partial upgrades as a future feature; it stops manufacturing the churn that makes them impossible. + +One narrower question remains, and it is the one worth testing: **is the generated tarball deterministic for identical input content across two different source revisions?** If source-watcher preserves modification times from its unpack of the source, the digest could vary build-to-build even for byte-identical files, and Option A would need a determinism fix upstream rather than a design change here. This is a tarball-reproducibility question, not an architecture question, and it does not need a Cozystack cluster: source-controller plus source-watcher on kind, two OCI artifacts differing in one directory, and a comparison of `status.artifact.digest` on an untouched `ExternalArtifact` across the two. The result belongs in this document before the design is locked. + +Option B remains the target for external catalogs, and the groundwork is favourable: `ArtifactGenerator.spec.sources[]` is **already a list with aliases** upstream, and the copy operations already address `@alias/path/**`. Cozystack always writes exactly one entry (`packagesource_reconciler.go:268-275`), so the change is confined to `reconcileArtifactGenerators`. The cost is object count — from 2 `OCIRepository` objects to roughly 100, each with a 5-minute poll interval, plus source-watcher's `emptyDir` unpack (`internal/fluxinstall/manifests/fluxcd.yaml:8287-8288`) rebuilding on every pod restart. That must be load-tested, not assumed. The recommended sequencing is therefore Option A for the first-party archive now, Option B when external catalogs land. A third structural coupling to resolve under either option: `package_reconciler.go:131` looks up the `PackageSource` **by the same name as the `Package`**. That 1:1 naming is fine if version lives as a field on those objects, and blocks anything that would need parallel objects per version. @@ -395,7 +401,7 @@ Either way, security fixes need an exception path: a patch release of the curren ## Testing -- **Conformance, in-tree and gating:** a synthetic package published at two versions; assert that upgrading the manifest from v1 to v2 reconciles exactly that package and leaves an untouched neighbour's HelmRelease revision unchanged. This is the acceptance test for the entire proposal, and it doubles as the [experiment](#4-the-versioned-pool--two-options-and-the-experiment-that-decides) that chooses Option A or B. +- **Conformance, in-tree and gating:** a synthetic package published at two versions; assert that upgrading the manifest from v1 to v2 reconciles exactly that package and leaves an untouched neighbour's `ExternalArtifact` digest and HelmRelease revision unchanged. This is the acceptance test for the entire proposal, and it subsumes the determinism check in rollout phase 2. - **CI gates:** content-changed-implies-bump, and schema-diff-classifies-bump, both run per PR over the package tree. - **Migration framework:** unit tests on the `cozy-lib` version helper including the string-vs-integer comparison at 10; per-package hook tests reusing the existing helm-unittest harness. - **Upgrade e2e:** a two-release chain (`2026.08` → `2026.09`) with a partial manifest diff, asserting untouched workloads are not restarted. The existing seed → upgrade → verify lane is the right host for this. @@ -407,7 +413,7 @@ Either way, security fixes need an exception path: a patch release of the curren The ordering is chosen so that each phase is independently valuable and independently revertible, and so that the riskiest item (the versioned pool) is preceded by the experiment that de-risks it. 1. **Cadence only.** Adopt the four-week train, publish the support window, cut `release-YYYY.MM` branches. No code changes. Two or three cycles of evidence before anything else depends on the cadence holding. -2. **Experiment and decision.** Run the `ExternalArtifact` revision experiment; record the result in this document; choose Option A or Option B. +2. **Determinism check.** Confirm on kind (source-controller + source-watcher only) that a generated artifact's digest is stable for identical input content across two source revisions. Option A is already the chosen path per [§4](#4-the-versioned-pool); this phase only establishes whether it needs an upstream reproducibility fix alongside it. 3. **Manifest, inert.** Publish the release manifest with all versions equal to the current platform version. Add `version` to the data model. Nothing resolves differently. Add the two CI gates. 4. **ApplicationDefinition consolidation.** One coordinated pass folding this proposal's fields with #39's, #6's, and #3448's, before external catalogs depend on the shape. 5. **Migration framework.** `cozy-lib` migration helper; port the four existing ad-hoc hooks onto it; new migrations are authored per-package; the global lane is frozen except for cross-cutting cases. @@ -418,7 +424,7 @@ The ordering is chosen so that each phase is independently valuable and independ ## Open questions -- **Option A or Option B** — resolved by the experiment in phase 2. Everything else in the design is stable across the answer; only the implementation site changes. +- **Is the generated artifact tarball reproducible?** Option A is chosen ([§4](#4-the-versioned-pool)), but if source-watcher's tarball carries unpack-time modification times, identical content can yield differing digests and partial upgrades need an upstream determinism fix alongside this work. Checkable on kind; the answer changes the implementation's dependencies, not the design. - **Manifest kind and home.** A CRD applied to the cluster, a plain OCI artifact read by tooling, or both? #23's `TapIndex` cache is the obvious reader either way. - **How does the CalVer name relate to the existing `v1.x` stream?** Is `2026.08` a rename of the same stream, or does a final `v1.x` release announce the switch? What do existing release branches and backport automation do at the boundary? - **Support window length.** Three releases? Six? Security-only tail? This must be answered before phase 1, not after. @@ -458,6 +464,8 @@ Collected from `main` on 2026-07-27 (post-`v1.6.0`). Line references are to that | Existing per-package migration hooks | 4 | seaweedfs, etcd-operator, dashboard, vm-disk/vm-instance | | `cozypkg` size / commits / documentation | 1811 lines / 3 commits / 0 lines of docs | `cmd/cozypkg/`, `git log` | | Core code churn, 12 months | ~620 commits across `internal/`, `pkg/`, `api/` | `git log --since='12 months ago'` | +| Generated artifact revision derivation | content digest — the field is optional and cozystack never sets it | `fluxcd.yaml:522-528` (CRD doc), `packagesource_reconciler.go:241-244` (no `Revision`) | +| Consequence | the delivery layer would already skip unchanged packages; the per-release chart-version rewrite is what guarantees none are ever unchanged | `packages/core/installer/Makefile` | ## Appendix B — unrelated defects found while surveying From 0171b2be063be218a96d4c432373d07b5d961db8 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Thu, 27 Aug 2026 14:38:52 +0500 Subject: [PATCH 03/13] docs(cozystack-as-a-distribution): refresh related-proposal statuses MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #18 and #23 were merged on 2026-08-24 and are therefore accepted under the approval process, not drafts. Record that, link the merged documents, and note that #23 restates #18's position on per-package pinning — so section 8 now differs from two accepted documents rather than one draft. Also scope #18's non-goal correctly: per-package pinning is out of scope for its Phase 1, which is a sequencing statement rather than a permanent exclusion, and describe #23's credential threading the way the merged text does — a secretRef on the Flux source a tap already creates, no CRD change. Assisted-By: Claude Signed-off-by: Myasnikov Daniil --- design-proposals/cozystack-as-a-distribution/README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/design-proposals/cozystack-as-a-distribution/README.md b/design-proposals/cozystack-as-a-distribution/README.md index 502bf8e..85d2049 100644 --- a/design-proposals/cozystack-as-a-distribution/README.md +++ b/design-proposals/cozystack-as-a-distribution/README.md @@ -22,14 +22,14 @@ This proposal is deliberately a joining piece. Several accepted or in-flight pro | Proposal | What it establishes | How this proposal relates | |---|---|---| | [community#43](https://github.com/cozystack/community/pull/43) — Out-of-tree application catalogs (`@lllamnyp`, Review) | Three application tiers (platform / curated catalog / external catalog), catalog repositories as OCI artifacts, e2e cost as the driver, a declared minimum platform version checked at `PackageSource` reconciliation | **Supplies the version axis #43 assumes.** #43 defines the *component structure* of the distribution (Debian's main/universe split); this proposal defines what a version means inside it and where the declared minimum platform version comes from. #43 is the stronger near-term motivator and should not wait on this | -| [community#18](https://github.com/cozystack/community/pull/18) — Cozymarketplace (`@kvaps`, Draft) | Repository-as-unit, a krew-style meta-index, a repository is one versioned OCI artifact, `cozypkg` repository commands | **Adopted as-is for external repositories; extended, not replaced, for the platform's own.** #18 makes per-package pinning a non-goal, which is the right call for the problem #18 is solving. This proposal reaches a different conclusion for first-party packages from a different requirement — see [§8](#8-repository-as-unit-versus-package-as-unit) | +| [community#18](https://github.com/cozystack/community/pull/18) — Cozymarketplace (`@kvaps`, **accepted**, merged 2026-08-24 as [`design-proposals/cozymarketplace`](https://github.com/cozystack/community/blob/main/design-proposals/cozymarketplace/README.md)) | Repository-as-unit, a krew-style meta-index, a repository is one versioned OCI artifact, `cozypkg` repository commands | **Adopted as-is for external repositories; extended, not replaced, for the platform's own.** #18 makes per-package pinning a non-goal **for its Phase 1**, which is the right call for the problem #18 is solving; it is a sequencing statement, not a permanent exclusion. This proposal reaches a different conclusion for first-party packages from a different requirement — see [§8](#8-repository-as-unit-versus-package-as-unit) | | [community#12](https://github.com/cozystack/community/pull/12) — Community package index and `cozypkg` authoring (`@kvaps`, `@IvanHunters`, closed 2026-07-16) | `cozypkg tap` / `init` / `push` / `search`, metadata-only index, `community.` name prefixing, an optional expected-signing-identity per entry | **Its package-level axis is revived here, on a different justification.** #12 was package-centric; #18 superseded it with a repository-centric model and #12 was closed. That supersession is sound on ecosystem grounds. This proposal re-derives the package-level axis from platform release engineering instead, and reuses #12's `cozypkg` surface and index-entry shape rather than inventing new ones | -| [community#23](https://github.com/cozystack/community/pull/23) — Cozymarketplace supplementary (`@IvanHunters`, Draft) | Marketplace endpoints in `cozystack-api`, a `TapIndex` cache, `secretRef` threading for private taps, `cozypkg validate` | **Consumed unchanged.** The `TapIndex` cache is the natural home for the release manifest / index reader described below | +| [community#23](https://github.com/cozystack/community/pull/23) — Cozymarketplace supplementary (`@IvanHunters`, **accepted**, merged 2026-08-24 as [`design-proposals/cozymarketplace-supplementary`](https://github.com/cozystack/community/blob/main/design-proposals/cozymarketplace-supplementary/README.md)) | Marketplace endpoints in `cozystack-api`, a `TapIndex` cache, a pull credential set as `spec.secretRef` on the Flux source a tap already creates (no CRD change), `cozypkg validate` plus a two-lane index CI gate | **Consumed unchanged.** The `TapIndex` cache is the natural home for the release manifest / index reader described below. Note that #23 restates #18's position — "per-package version pinning remains out of scope, siding with `#18`" — so this proposal now differs from two accepted documents rather than one draft; see [§8](#8-repository-as-unit-versus-package-as-unit) | | [community#6](https://github.com/cozystack/community/pull/6) — ApplicationDefinition multi-version conversion (`@kvaps`, Draft) | `versions[]` with a storage version, `to`/`from` conversion templates, a `_version` stamp, a background migration controller | **Orthogonal axis, and the thing that makes semver honest.** #6 versions the *API surface* of an app; this proposal versions the *package*. Their relationship is what gives "breaking change" a testable definition — see [What MAJOR means](#what-major-means) | | [community#39](https://github.com/cozystack/community/pull/39) — Fold `extra` into `apps` (`@myasnikovdaniil`, Draft) | Retires the `extra` bucket; visibility / cardinality / protection / capability as declarative `ApplicationDefinition` fields | **Same direction, one tier down.** #39 removes a directory-as-metadata convention by moving the metadata onto the ApplicationDefinition; this proposal does the same for the version. Together they turn ApplicationDefinition into the package's control file | | [cozystack#3448](https://github.com/cozystack/cozystack/pull/3448) — ApplicationGroupDefinition (MVP) | Dynamic API-group registration, reserved-namespace validation, restart-based pickup via the `cozystack.io/config-hash` rollout | **Prerequisite for tier 2, and the proof that the runtime is already data-driven.** The mechanism it extends is the reason per-package delivery is feasible at all | | [community#21](https://github.com/cozystack/community/pull/21) — Self-hosted in-cluster registry for air-gap (`@gecube`, Draft) | Offline bundle, in-cluster registry as source of truth | **Affected, and improved.** A per-package pool changes what an offline bundle contains; a release manifest is exactly the mirror list an air-gapped site needs. Flagged as an open question rather than solved here | -| [community#25](https://github.com/cozystack/community/pull/25), [community#33](https://github.com/cozystack/community/pull/33) | Per-cluster etcd; ComputePlane as an operator-owned module | Downstream consumers of the tiering; not blocked by this proposal | +| [community#25](https://github.com/cozystack/community/pull/25) (Draft), [community#33](https://github.com/cozystack/community/pull/33) (**accepted**, merged 2026-07-18) | Per-cluster etcd; ComputePlane as an operator-owned module | Downstream consumers of the tiering; not blocked by this proposal | **Deliberately out of scope:** the content of any specific tier assignment (that is #43's call), the API-conversion mechanism (that is #6's), the dashboard's marketplace information architecture (#18/#23), and anything about how Talos or the installer bootstraps a cluster. @@ -296,7 +296,7 @@ Two existing defects to fix while touching it: `add -f` on an existing Package f This is the one place where this proposal reaches a different conclusion from an existing one, so it is set out in full rather than elided. -**The history matters, because it is a considered position and not an oversight.** [#12](https://github.com/cozystack/community/pull/12) (2026-05-26) was package-centric: publish, index, and install individual packages. [#18](https://github.com/cozystack/community/pull/18) (2026-06-23) proposed the repository as the unit instead, listing the package-centric model under Alternatives considered and rejecting it on the grounds that a thematic repository carries a "tested together" guarantee that a loose package catalog does not. #12 was then closed on 2026-07-16. For the problem #18 addresses — how a community publishes coherent, mutually-tested sets of applications that Cozystack maintainers have not reviewed — **that reasoning is correct, and this proposal adopts #18's model unchanged for that case.** A third-party repository is authored and tested as a set, its author owes Cozystack no compatibility guarantee, and a repository-level tag costs nothing because the OCI artifact already exists. +**The history matters, because it is a considered position and not an oversight.** [#12](https://github.com/cozystack/community/pull/12) (2026-05-26) was package-centric: publish, index, and install individual packages. [#18](https://github.com/cozystack/community/pull/18) (2026-06-23) proposed the repository as the unit instead, listing the package-centric model under Alternatives considered and rejecting it on the grounds that a thematic repository carries a "tested together" guarantee that a loose package catalog does not. #12 was then closed on 2026-07-16, and both #18 and #23 were merged on 2026-08-24 — which under the [approval process](https://github.com/cozystack/community/blob/main/design-proposals/README.md#approval-process) makes them accepted, not merely proposed. This proposal therefore argues against a settled position rather than a competing draft, and the burden is correspondingly higher. For the problem #18 addresses — how a community publishes coherent, mutually-tested sets of applications that Cozystack maintainers have not reviewed — **that reasoning is correct, and this proposal adopts #18's model unchanged for that case.** A third-party repository is authored and tested as a set, its author owes Cozystack no compatibility guarantee, and a repository-level tag costs nothing because the OCI artifact already exists. The different conclusion here comes from a requirement neither #12 nor #18 was scoped to weigh: **the platform's own release engineering.** #18's "tested together" argument is exactly the argument for a manifest — and a manifest that can only name whole repositories cannot express "these three packages moved and the other 155 did not". Shipping `2026.09` with the same core and three bumped apps is the entire value of the partial-upgrade goal, and it is unreachable if the finest addressable unit is the repository. So the disagreement is narrow and does not touch #18's thesis: it is about whether the *first-party archive* is one repository or many packages, not about how community repositories should work. @@ -385,7 +385,7 @@ Either way, security fixes need an exception path: a patch release of the curren - **External packages must never reach the global migration lane.** This is a hard boundary, not a guideline. The global lane is core-owned; a tapped repository ships per-package hooks only. - **Installing a repository stays a cluster-admin action,** as #43 and #18 both state: a catalog causes the platform to render and apply arbitrary Helm charts. - **A versioned pool makes signing worth standardising.** #43 lists artifact signing as an open question; with per-package artifacts the manifest is the natural place to carry expected signing identity per package, and #12 already anticipated an "expected signing identity" field in its index entries. -- **Private repositories** are handled by #23's `secretRef` threading; nothing here changes it. +- **Private repositories** are handled by #23's credential threading — a `Secret` referenced as `spec.secretRef` on the Flux source that `cozypkg tap` already creates, with no CRD change; nothing here changes it. - **No new tenant-facing surface.** Everything in this proposal is admin-facing or build-time. ## Failure and edge cases From 769bee3eecb8c1006ef512375dd2bc63a1d0b21e Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Thu, 27 Aug 2026 17:03:50 +0500 Subject: [PATCH 04/13] docs(cozystack-as-a-distribution): correct the churn mechanism, draw the core boundary MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The proposal's headline finding was wrong. Nothing rewrites Chart.yaml in the pool: `helm package --version` runs only in the `repo` target that builds _out/repos/, the neighbouring fix-charts target resets the field to 0.0.0, and the pool is a `flux push artifact` over the source tree — so all 164 charts ship `version: 0.0.0` and the version stamp never enters the artifact at all. What actually manufactures churn is a decorative version substring vendored inside each chart's image reference and rewritten at promotion by hack/promote-rewrite-tags.sh. Measured over v1.6.1..v1.6.2: of 164 packages, 129 are untouched, 9 changed chart source, 13 had an image digest move, and 13 moved with a byte-identical digest because only the tag string changed. Kubernetes resolves those references by digest and never reads the tag. So the churn is neither total nor unavoidable, and dropping the vendored tag becomes rollout phase 1 — cheap, independent of every design commitment here, and measurable on the next patch release. The tarball determinism question is answered rather than deferred: flux 2.8.6 normalises archive entries to epoch mtime and zero uid/gid, and building one tree twice is byte-identical. The residual is source-watcher's own re-tar, which is a code read rather than a cluster experiment. Section 1 now draws the core boundary the way Kubernetes draws its own — core is the mechanism, a package is anything that runs a workload, storage and networking included — and classifies the eleven charts that build from the root Go context against it. Two consequences are recorded: the delivery boundary is in the wrong place today, since the apiserver, controller and lineage webhook travel through the pool as ordinary packages and extracting them is the largest unscoped piece of work here; and because the Go tree stays one module by choice, a package's version cannot be derived from its image digest, which is an independent reason the digest must move to the manifest. Section 5 follows from that: PATCH and MINOR are derived from a package's own directory, MAJOR is declared, and CI checks the declaration is honest — reusing cmd/api-gate, which already computes that verdict for the API surface. Section 8 is reframed as inheritance rather than divergence. #18 scopes its exclusion of per-package pinning to Phase 1, so this is the Phase 2 that sentence leaves room for, and nothing here asks #18 or #23 to be amended. Assisted-By: Claude Signed-off-by: Myasnikov Daniil --- .../cozystack-as-a-distribution/README.md | 145 ++++++++++++++---- 1 file changed, 111 insertions(+), 34 deletions(-) diff --git a/design-proposals/cozystack-as-a-distribution/README.md b/design-proposals/cozystack-as-a-distribution/README.md index 85d2049..7611284 100644 --- a/design-proposals/cozystack-as-a-distribution/README.md +++ b/design-proposals/cozystack-as-a-distribution/README.md @@ -7,11 +7,13 @@ ## Overview -Cozystack has exactly one version number, and everything wears it. 160 of the 164 charts under `packages/` ship with `version: 0.0.0 # Placeholder, the actual version will be automatically set during the build process`, stamped at build time from `COZYSTACK_VERSION`; the whole `packages/` tree is pushed as a single OCI artifact; a single digest in `packages/core/installer/values.yaml` selects it; and one `OCIRepository` fans that digest out to 98 `PackageSource` objects. A Postgres chart fix and a change to the aggregated apiserver are, as far as delivery is concerned, the same event. +Cozystack has exactly one version number, and everything wears it. `COZYSTACK_VERSION` comes from `git describe --tags` in `hack/common-envs.mk` and every package Makefile reads it; the whole `packages/` tree is pushed as a single OCI artifact; a single digest in `packages/core/installer/values.yaml` selects it; and one `OCIRepository` fans that digest out to 98 `PackageSource` objects. A Postgres chart fix and a change to the aggregated apiserver are, as far as delivery is concerned, the same event. + +The charts themselves carry no version at all. All 164 under `packages/` ship `version: 0.0.0 # Placeholder, the actual version will be automatically set during the build process`, and nothing in the pool build ever replaces it: `helm package --version $(COZYSTACK_VERSION)` appears only in the `repo` target of `packages/{apps,system,extra,library}/Makefile`, which builds the separate Helm repository under `_out/repos/`, while the neighbouring `fix-charts` target *resets* the field to `0.0.0`. The pool is `flux push artifact --path=../../../packages` over the source tree, so what ships is `version: 0.0.0`, 164 times. The placeholder comment describes a build step that does not exist. This proposal replaces that single number with four, arranged the way an operating-system distribution arranges its own: a **core** on semantic versioning (the apiserver, controller, operator, CRDs, and the delivery contract itself), **packages** and **apps** each carrying their own semantic version, and a **distribution** — the thing users install and talk about — on calendar versioning, released monthly. A release stops being "the state of a git tree at a tag" and becomes what a distribution release has always been: a manifest that pins an exact set of component versions, tested together, shipped together, supported together. -The change is smaller than it sounds, because the platform is already most of the way there and nobody has been calling the pieces by their distribution names. `PackageSource` and `Package` are a package database. Bundles are metapackages. `helm.sh/resource-policy: keep` on the `Package` CRs means the installed set already diverges from the shipped set — that is `dpkg --get-selections`, implemented by accident. `cozypkg` is `apt` with the version, repository, and index code removed. The per-package pre-upgrade migration hook exists and is used by four packages already. And the delivery layer already does the hardest part: generated artifact revisions are content-derived, so helm-controller already skips a package whose content did not change — the per-release rewrite of every `Chart.yaml` version is what guarantees no package is ever unchanged. What is genuinely missing is narrow: a versioned pool, an index, and eventually a version field on the source reference. This proposal specifies those and then connects the in-flight proposals that each assume some part of them. +The change is smaller than it sounds, because the platform is already most of the way there and nobody has been calling the pieces by their distribution names. `PackageSource` and `Package` are a package database. Bundles are metapackages. `helm.sh/resource-policy: keep` on the `Package` CRs means the installed set already diverges from the shipped set — that is `dpkg --get-selections`, implemented by accident. `cozypkg` is `apt` with the version, repository, and index code removed. The per-package pre-upgrade migration hook exists and is used by four packages already. And the delivery layer already does the hardest part: generated artifact revisions are content-derived, so helm-controller already skips a package whose content did not change. What defeats it is narrower and more embarrassing than a version scheme — a cosmetic version string vendored inside each chart's image reference, rewritten on every release even for images that were not rebuilt (see [§4](#4-the-versioned-pool)). What is genuinely missing is narrow: a versioned pool, an index, and eventually a version field on the source reference. This proposal specifies those and then connects the in-flight proposals that each assume some part of them. The case is made from platform release engineering rather than from ecosystem ambition — the costs it addresses are paid every release cycle today, in a repository with no external catalogs at all. That framing is deliberate and is set out in [Why this is a release-engineering proposal](#why-this-is-a-release-engineering-proposal-not-a-marketplace-proposal), because it determines build order: the internal requirement can be satisfied with no discovery surface whatsoever, while a discovery surface cannot be satisfied without it. @@ -22,7 +24,7 @@ This proposal is deliberately a joining piece. Several accepted or in-flight pro | Proposal | What it establishes | How this proposal relates | |---|---|---| | [community#43](https://github.com/cozystack/community/pull/43) — Out-of-tree application catalogs (`@lllamnyp`, Review) | Three application tiers (platform / curated catalog / external catalog), catalog repositories as OCI artifacts, e2e cost as the driver, a declared minimum platform version checked at `PackageSource` reconciliation | **Supplies the version axis #43 assumes.** #43 defines the *component structure* of the distribution (Debian's main/universe split); this proposal defines what a version means inside it and where the declared minimum platform version comes from. #43 is the stronger near-term motivator and should not wait on this | -| [community#18](https://github.com/cozystack/community/pull/18) — Cozymarketplace (`@kvaps`, **accepted**, merged 2026-08-24 as [`design-proposals/cozymarketplace`](https://github.com/cozystack/community/blob/main/design-proposals/cozymarketplace/README.md)) | Repository-as-unit, a krew-style meta-index, a repository is one versioned OCI artifact, `cozypkg` repository commands | **Adopted as-is for external repositories; extended, not replaced, for the platform's own.** #18 makes per-package pinning a non-goal **for its Phase 1**, which is the right call for the problem #18 is solving; it is a sequencing statement, not a permanent exclusion. This proposal reaches a different conclusion for first-party packages from a different requirement — see [§8](#8-repository-as-unit-versus-package-as-unit) | +| [community#18](https://github.com/cozystack/community/pull/18) — Cozymarketplace (`@kvaps`, **accepted**, merged 2026-08-24 as [`design-proposals/cozymarketplace`](https://github.com/cozystack/community/blob/main/design-proposals/cozymarketplace/README.md)) | Repository-as-unit, a krew-style meta-index, a repository is one versioned OCI artifact, `cozypkg` repository commands | **Adopted as-is for external repositories; extended, not replaced, for the platform's own.** #18 makes per-package pinning a non-goal **for its Phase 1**, which is the right call for the problem #18 is solving; it is a sequencing statement, not a permanent exclusion. This proposal is the Phase 2 it leaves room for, scoped to the first-party archive — see [§8](#8-repository-as-unit-versus-package-as-unit) | | [community#12](https://github.com/cozystack/community/pull/12) — Community package index and `cozypkg` authoring (`@kvaps`, `@IvanHunters`, closed 2026-07-16) | `cozypkg tap` / `init` / `push` / `search`, metadata-only index, `community.` name prefixing, an optional expected-signing-identity per entry | **Its package-level axis is revived here, on a different justification.** #12 was package-centric; #18 superseded it with a repository-centric model and #12 was closed. That supersession is sound on ecosystem grounds. This proposal re-derives the package-level axis from platform release engineering instead, and reuses #12's `cozypkg` surface and index-entry shape rather than inventing new ones | | [community#23](https://github.com/cozystack/community/pull/23) — Cozymarketplace supplementary (`@IvanHunters`, **accepted**, merged 2026-08-24 as [`design-proposals/cozymarketplace-supplementary`](https://github.com/cozystack/community/blob/main/design-proposals/cozymarketplace-supplementary/README.md)) | Marketplace endpoints in `cozystack-api`, a `TapIndex` cache, a pull credential set as `spec.secretRef` on the Flux source a tap already creates (no CRD change), `cozypkg validate` plus a two-lane index CI gate | **Consumed unchanged.** The `TapIndex` cache is the natural home for the release manifest / index reader described below. Note that #23 restates #18's position — "per-package version pinning remains out of scope, siding with `#18`" — so this proposal now differs from two accepted documents rather than one draft; see [§8](#8-repository-as-unit-versus-package-as-unit) | | [community#6](https://github.com/cozystack/community/pull/6) — ApplicationDefinition multi-version conversion (`@kvaps`, Draft) | `versions[]` with a storage version, `to`/`from` conversion templates, a `_version` stamp, a background migration controller | **Orthogonal axis, and the thing that makes semver honest.** #6 versions the *API surface* of an app; this proposal versions the *package*. Their relationship is what gives "breaking change" a testable definition — see [What MAJOR means](#what-major-means) | @@ -85,7 +87,7 @@ One counter-fact to handle: `api/apps/v1alpha1/` is a **separate Go module** (`a ### The problem -- **Every release moves every chart, and the version stamp is why.** Because chart versions are rewritten from `COZYSTACK_VERSION` before the pool is pushed, a release changes the version of 160 charts whether or not their content changed — and since generated artifact revisions are content-derived (see [§4](#4-the-versioned-pool)), that stamp is what forces every HelmRelease to upgrade. The delivery machinery would already skip unchanged packages; the single version number is what prevents any package from ever being unchanged. Users upgrading for a Postgres fix take the whole platform with them, and upgrade windows, blast radius, and rollback all scale with the size of the tree rather than the size of the change. +- **A release moves packages that did not change, and a decorative tag string is why.** Measured across `v1.6.1..v1.6.2`, 35 of the 164 packages have any changed file at all: 9 changed chart source, 13 had an image digest move, and **13 moved because the version substring inside an image reference was rewritten while the digest stayed byte-identical.** The remaining 129 are untouched. Since generated artifact revisions are content-derived (see [§4](#4-the-versioned-pool)), each of those 13 nevertheless gets a new artifact digest and a HelmRelease upgrade, for a change Kubernetes never reads — it pulls by digest. So the churn is neither total nor unavoidable; it is manufactured by one line of cosmetics per chart, and removing it is a smaller change than anything else in this proposal. - **Breaking changes are held hostage.** A breaking change in one app forces either a platform major (which frightens users away from an upgrade that is mostly bug fixes) or an indefinite delay. There is no way to say "MongoDB 4.0 is breaking, the platform is not". - **A package fix cannot ship without a platform release.** Today the only route to a released cluster is a backport, and the backport bot fails silently on conflict and imports whole files for files absent on the target branch. Security fixes inherit that latency. - **Nothing can be tested or supported at package granularity,** because nothing *is* a package at release granularity. #43 wants catalog repositories with their own CI against a matrix of supported Cozystack releases; there is no version to put in that matrix except the whole platform's. @@ -96,7 +98,7 @@ One counter-fact to handle: `api/apps/v1alpha1/` is a **separate Go module** (`a Per-package versioning has come up before in an ecosystem context — publishing, discovery, letting third parties ship apps — and in that context it was reasonably set aside. This proposal deliberately does not argue from ecosystem ambition, a roadmap item, or what other platforms have. It argues from four things that hurt today, in a repository with no external catalogs at all: 1. **A fix cannot reach a released cluster without a platform release.** The only route today is a backport, and the backport automation fails silently on conflict and imports whole files when the target branch lacks the file being modified. Security fixes inherit that latency and that failure mode. -2. **Every upgrade is a full-platform upgrade.** A month whose substance is three bug fixes still moves 160 chart versions, re-reconciles the tree, and asks operators for a maintenance window sized to the whole platform rather than to the change. +2. **An upgrade is sized by the release, not by the change.** A patch release whose substance is nine chart-source fixes moves 35 packages, 13 of them for a rewritten tag string alone ([§4](#4-the-versioned-pool)), and the operator is asked for a maintenance window sized to the release rather than to the change — because nothing in the release tells them which 9 mattered. 3. **Breaking changes are structurally discouraged.** With one version, a breaking change in one app is a platform-level event. The rational response is to defer it, and deferred schema debt is why [#6](https://github.com/cozystack/community/pull/6) exists at all — the shapes it wants to fix (`users` as a map, `postgresql.parameters.max_connections`) have been wrong for a long time and stayed wrong because there was no way to charge the cost to one app. 4. **Release readiness is all-or-nothing.** `v1.6.0` needed four release candidates and a NO-GO on rc.1 for upgrade-only blockers in two components. With a single version, two blocked components block everything; with a manifest, they hold at their previous versions and the train ships. @@ -126,7 +128,7 @@ None of those four require a marketplace, a community index, or a single externa | Tier | Contents | Versioning | Cadence | |---|---|---|---| -| **Core** | `cozystack-api`, `cozystack-controller`, `cozystack-operator`, the `cozystack.io` CRDs, and the delivery contract (`PackageSource`, `Package`, `ApplicationDefinition`, `ApplicationGroupDefinition`) | SemVer | On its own, as needed | +| **Core** | The mechanism: the aggregated apiserver, the controllers that reconcile Cozystack's own API, the tenant machinery, the package and application system, the `cozystack.io` CRDs, and the delivery contract (`PackageSource`, `Package`, `ApplicationDefinition`, `ApplicationGroupDefinition`) | SemVer | On its own, as needed | | **Packages** | Platform components — operators, CSI/CNI, monitoring, ingress; today's `packages/system` | SemVer, Cozystack's own, not upstream's | Continuous, released from `main` | | **Apps** | User-facing managed applications; today's `packages/apps` (and `packages/extra` after #39) | SemVer | Continuous, released from `main` | | **Distribution** | The thing users install and name | CalVer `YYYY.MM` | Monthly | @@ -135,6 +137,36 @@ Mechanically, packages and apps are the **same object** — both are `PackageSou Core is the one genuine exception, and it must be defined by **contract, not by binary**. If core's version tracks the Go code, a controller bug fix and a CRD field removal both read as "core changed" and the number communicates nothing. Core's semver describes the compatibility of the CRD groups, the `PackageSource`/`Package`/`ApplicationDefinition` semantics, and the operator's artifact-resolution behaviour. The binaries are implementation. +#### Where the line falls + +The governing analogy is Kubernetes itself: the apiserver, the controller manager and the API machinery *are* Kubernetes, while the CNI, the CSI driver and the ingress controller are things that run on it, however indispensable. Applied here, **core is the mechanism and a package is anything that runs a workload** — including storage and networking, which are load-bearing but are not the platform. + +Eleven charts build their image from the repository root with `COPY api pkg cmd internal`, which makes them the set where the line is not obvious. Classified: + +| Chart or binary | What it does | Tier | +|---|---|---| +| `cozystack-api` | aggregated apiserver | **core** | +| `cozystack-controller` | reconciles `Tenant` and `Application` | **core** | +| `cozystack-operator` (ships in `packages/core/installer`) | `PackageSource` / `Package` | **core** | +| `lineage-controller-webhook` | admits against `api/v1alpha1`, Cozystack's own CRDs | **core** | +| `cozypkg` | the package manager (released as a binary asset, not an image) | **core** | +| `packages/core/platform` | bundles, sources, the shape of the platform | **core** | +| `flux-plunger`, `flux-shard-operator` | keep Flux delivering packages | **core**, but see the open question below | +| `backup-controller`, `backupstrategy-controller` | own API group `api/backups/*`; a backup service | package | +| `securitygroup-controller`, `kubeovn-plunger` | networking | package | + +Everything else under `packages/{system,apps,extra}` is a package without argument. + +#### Two consequences the rest of this proposal depends on + +**The delivery boundary is in the wrong place today, and moving it is real work.** Only two things currently ship outside the package pool: the CRDs, concatenated from `internal/crdinstall/manifests/*.yaml` into `_out/assets/cozystack-crds.yaml`, and the operator, rendered from `packages/core/installer` by the root `manifests` target. Flux itself comes from `internal/fluxinstall/manifests/fluxcd.yaml`. Everything else in the table above — the apiserver, the controller, the lineage webhook — travels through the pool as ordinary `packages/system/*` charts, on the same path as Cilium. Declaring them core therefore means moving three charts from the pool into the bootstrap lane, which changes how they are installed, upgraded and rolled back. This is the largest single piece of unscoped work in the proposal and it should be planned as its own change, not folded into the versioning work. + +A naming collision comes with it: `packages/core/` today holds `flux-aio`, `installer`, `platform`, `talos` and `testing`, of which only `platform` and part of `installer` are core in the sense used here. The directory name will have to give way, most likely by the same argument #39 makes — the tier is a declared field, not a path. + +**The Go tree stays one unit, so package versions must not be derived from image digests.** All eleven charts build from one `go.mod` and one build context, so a change anywhere in `internal/` invalidates the cache for every one of them and moves every digest. Splitting the module is explicitly not proposed: the coupling is real but the cost of severing it exceeds the benefit, and core being a single Go module is consistent with core being a single SemVer unit. + +The consequence is that a package's version cannot be a function of its image digest, because that digest moves for reasons belonging to a different tier. `backup-controller` is a package whose image is rebuilt whenever the apiserver changes. Two things follow, and both are already required for other reasons: the digest must not live inside the chart (see [§4](#4-the-versioned-pool) — it moves to the release manifest, which is also what removes the tag-rewrite churn), and a package's version must be derived from changes to *its own* directory rather than from its build output. + ### 2. The distribution mapping Naming the analogy precisely is useful, because it makes each missing piece obvious and tells reviewers which prior art to argue from. @@ -207,9 +239,38 @@ An earlier draft of this proposal treated the choice as gated on an unknown — The field is optional, and cozystack never sets it: `reconcileArtifactGenerators` constructs `OutputArtifact{Name, Copy}` at `internal/operator/packagesource_reconciler.go:241-244`, and the string `Revision` does not appear anywhere in that file. The `@` pattern constrains the field *when set*; it is not evidence that revisions track the source. So generated artifact revisions are already content-derived, on every cluster running today. -**This reframes the whole proposal, and it is the most important finding in it.** Content-derived revisions mean helm-controller already ignores a package whose content did not change. The reason every package nevertheless upgrades on every release is the version stamp itself: `packages/core/installer/Makefile` rewrites `Chart.yaml` from `COZYSTACK_VERSION` for all 160 charts before the pool is pushed, so every chart's content changes on every release, so every artifact digest changes, so every HelmRelease upgrades. **The single global version number is not merely a labelling problem — it is the direct mechanical cause of full-platform upgrade churn.** Giving charts real, stable versions does not merely enable partial upgrades as a future feature; it stops manufacturing the churn that makes them impossible. +**This reframes the whole proposal.** Content-derived revisions mean helm-controller already ignores a package whose content did not change — so the question is not how to make partial upgrades possible, but what is currently preventing them. It is not the chart version, which never enters the pool (see [Overview](#overview)). It is a decorative version string vendored inside each chart's image reference. -One narrower question remains, and it is the one worth testing: **is the generated tarball deterministic for identical input content across two different source revisions?** If source-watcher preserves modification times from its unpack of the source, the digest could vary build-to-build even for byte-identical files, and Option A would need a determinism fix upstream rather than a design change here. This is a tarball-reproducibility question, not an architecture question, and it does not need a Cozystack cluster: source-controller plus source-watcher on kind, two OCI artifacts differing in one directory, and a comparison of `status.artifact.digest` on an untouched `ExternalArtifact` across the two. The result belongs in this document before the design is locked. +#### What actually manufactures the churn + +Every package Makefile writes its built image back into its own chart as `:$(IMAGE_TAG)@sha256:…`, and at promotion `hack/promote-rewrite-tags.sh` rewrites the rc substring to the stable one across every ref-bearing file — `packages/*/*/values.yaml`, `packages/*/*/images/*.tag`, and the files declared in `hack/lib/image-refs.sh`. Promotion deliberately does not rebuild; the script's own header says "only the cosmetic tag string moves from `1.6.0-rc.4` to `1.6.0`". The result, from the real `v1.6.1..v1.6.2` diff: + +``` +packages/system/metallb/values.yaml +- tag: v1.6.1@sha256:9d8ba76cdb9c7c6221334ad05d706dee22b138b3e90c1fe8fc884925b7480c02 ++ tag: v1.6.2@sha256:9d8ba76cdb9c7c6221334ad05d706dee22b138b3e90c1fe8fc884925b7480c02 +``` + +Identical digest, different file. Kubernetes resolves this reference by digest and never reads the tag, but the chart's bytes changed, so its generated artifact digest changed, so its HelmRelease upgraded. Classified across the whole tree for that release: + +| `v1.6.1` → `v1.6.2` | packages | +|---|---| +| untouched | **129** | +| tag string only, image digest unchanged | **13** | +| image digest moved (rebuild) | 13 | +| chart source changed | 9 | + +Two things follow. First, "every release moves every chart" was never true: a patch release moves 35 of 164, and a minor (`v1.5.4` → `v1.6.0`) moves 92. Second, 13 of those 35 move for a string no runtime reads. + +**The fix is smaller than any other item in this proposal and independent of all of them: stop vendoring the tag.** Write `ghcr.io/cozystack/cozystack/metallb@sha256:…` and nothing else. The tag continues to exist in the registry — `hack/promote-retag.sh` pushes it — and the release manifest names it, so nothing that a human or a mirror needs is lost; only the copy that sits inside the chart and forces a reconcile goes away. `hack/promote-rewrite-tags.sh` and its bats suite are deleted outright, and the rc-leftover scan in `hack/verify-promoted-packages.sh` has nothing left to scan for. This is worth doing on its own merits whatever happens to the rest of the proposal, and its effect is measurable on the very next patch release. + +#### The tarball determinism question is answered + +An earlier draft listed as an open question whether the generated tarball is reproducible for identical input content, on the theory that unpack-time modification times could vary the digest. **It does not.** Flux normalises the archive: entries are written with mtime `1970-01-01`, uid/gid `0`, and fixed modes. Building the same directory twice with `flux build artifact` — pinned at 2.8.6, the version all three release workflows install — produces byte-identical output, `touch` in between included. + +One residual, stated precisely rather than waved away: that exercises the `flux` CLI's archive path, not source-watcher's re-tar inside `ArtifactGenerator`. The two share the upstream `fluxcd/pkg` archive code, so the remaining check is reading that code path rather than standing up a cluster, and rollout phase 3 shrinks accordingly. + +A related loose end worth fixing in the same pass: `flux push artifact --reproducible` exists — it fixes the OCI created-timestamp at epoch — and `packages/core/installer/Makefile:37` does not pass it. That affects the pool manifest's own digest, not the per-component artifacts, so it does not cause the churn above; it is simply free determinism that is currently declined. Option B remains the target for external catalogs, and the groundwork is favourable: `ArtifactGenerator.spec.sources[]` is **already a list with aliases** upstream, and the copy operations already address `@alias/path/**`. Cozystack always writes exactly one entry (`packagesource_reconciler.go:268-275`), so the change is confined to `reconcileArtifactGenerators`. The cost is object count — from 2 `OCIRepository` objects to roughly 100, each with a 5-minute poll interval, plus source-watcher's `emptyDir` unpack (`internal/fluxinstall/manifests/fluxcd.yaml:8287-8288`) rebuilding on every pod restart. That must be load-tested, not assumed. The recommended sequencing is therefore Option A for the first-party archive now, Option B when external catalogs land. @@ -225,12 +286,20 @@ SemVer without a written rule is noise, and this is the item most likely to quie | **MINOR** | Additive schema only (new optional keys with defaults); no removed or retyped keys; no immutable-field or selector change | Upgrade in place; may ship a per-package migration; release-note entry | | **MAJOR** | Any of: a removed or retyped values key; a change of storage version under #6; a workload rename or immutable-field change requiring adoption or recreation; a change to a capability the package provides to others (#39) | Requires a conversion path (#6) or a documented manual action; may not be crossed by a partial upgrade without an explicit gate | -Two CI gates make this real, and both must ship in the same phase as the version numbers: +#### Computed, not hand-written + +Maintaining 164 `Chart.yaml` versions by hand is not viable, so PATCH and MINOR are **derived**: a package's version is a function of the changes to its own directory since the last release, computed at build time. That keeps the number honest for free and removes an entire class of "forgot to bump" review comments. + +What a computed number cannot do is recognise a breaking change — no diff of files tells you that removing a values key strands existing clusters. **MAJOR is therefore declared, not derived.** The author states it, and CI's job is to check the declaration is honest rather than to produce it. + +Two CI gates make that real, and both must ship in the same phase as the version numbers: + +1. **Content changed implies version moved.** A package whose rendered output differs from the previous release without a version change fails the build. With derivation this is close to tautological, which is the point — it is the check that derivation is actually wired up. Without it, an unchanged chart version over changed content becomes an OCI/Flux cache-poisoning bug. +2. **Schema-diff classifies the bump.** Compare the generated OpenAPI schema against the previous version and assert the declared bump is at least as large as the diff implies. Removal or retyping demands MAJOR, and a package that removed a key while declaring MINOR fails. -1. **Content changed implies version bumped.** A package whose rendered output differs from the previous release without a version bump fails the build. Without this, an unchanged chart version over changed content becomes an OCI/Flux cache-poisoning bug. -2. **Schema-diff classifies the bump.** Compare the generated OpenAPI schema against the previous version and assert the bump is at least as large as the diff implies. Removal or retyping demands MAJOR. +The prior art for gate 2 is already in the tree and should be reused rather than re-invented: `cmd/api-gate` compares the Cozystack API surface across two checkouts, reports whether the change is "sizeable" — a new group, a new resource, or a break to an existing one — and CI turns that verdict into a required review from a designated API owner. That is exactly the shape wanted here, one tier down and with the verdict compared against a declaration instead of routed to a human. -The second gate is where #6 and this proposal meet: #6 gives an app the ability to *survive* a storage-form change, and this proposal gives it the number that *advertises* one. A MAJOR without a `to`/`from` pair is a manual-action release; with one, it is transparent. +Gate 2 is also where #6 and this proposal meet: #6 gives an app the ability to *survive* a storage-form change, and this proposal gives it the number that *advertises* one. A MAJOR without a `to`/`from` pair is a manual-action release; with one, it is transparent. ### 6. Migrations: per-package, generalising a pattern that already exists @@ -294,15 +363,15 @@ Two existing defects to fix while touching it: `add -f` on an existing Package f ### 8. Repository-as-unit versus package-as-unit -This is the one place where this proposal reaches a different conclusion from an existing one, so it is set out in full rather than elided. +This is the one place where this proposal could read as contradicting an accepted one, so it is set out in full rather than elided. The short version: it does not contradict #18, it continues it. #18 scopes its exclusion of per-package pinning to **Phase 1** — "out of scope for Phase 1; there is no per-package pinning today and the design proceeds without it" — which is a statement about sequencing, not a permanent property of the model. This proposal is the Phase 2 that sentence leaves room for, and it inherits #18's model wherever #18 has one. **The history matters, because it is a considered position and not an oversight.** [#12](https://github.com/cozystack/community/pull/12) (2026-05-26) was package-centric: publish, index, and install individual packages. [#18](https://github.com/cozystack/community/pull/18) (2026-06-23) proposed the repository as the unit instead, listing the package-centric model under Alternatives considered and rejecting it on the grounds that a thematic repository carries a "tested together" guarantee that a loose package catalog does not. #12 was then closed on 2026-07-16, and both #18 and #23 were merged on 2026-08-24 — which under the [approval process](https://github.com/cozystack/community/blob/main/design-proposals/README.md#approval-process) makes them accepted, not merely proposed. This proposal therefore argues against a settled position rather than a competing draft, and the burden is correspondingly higher. For the problem #18 addresses — how a community publishes coherent, mutually-tested sets of applications that Cozystack maintainers have not reviewed — **that reasoning is correct, and this proposal adopts #18's model unchanged for that case.** A third-party repository is authored and tested as a set, its author owes Cozystack no compatibility guarantee, and a repository-level tag costs nothing because the OCI artifact already exists. -The different conclusion here comes from a requirement neither #12 nor #18 was scoped to weigh: **the platform's own release engineering.** #18's "tested together" argument is exactly the argument for a manifest — and a manifest that can only name whole repositories cannot express "these three packages moved and the other 155 did not". Shipping `2026.09` with the same core and three bumped apps is the entire value of the partial-upgrade goal, and it is unreachable if the finest addressable unit is the repository. So the disagreement is narrow and does not touch #18's thesis: it is about whether the *first-party archive* is one repository or many packages, not about how community repositories should work. +The extension proposed here comes from a requirement neither #12 nor #18 was scoped to weigh: **the platform's own release engineering.** #18's "tested together" argument is exactly the argument for a manifest — and a manifest that can only name whole repositories cannot express "these three packages moved and the other 155 did not". Shipping `2026.09` with the same core and three bumped apps is the entire value of the partial-upgrade goal, and it is unreachable if the finest addressable unit is the repository. So the divergence is narrow and does not touch #18's thesis: it is about whether the *first-party archive* is one repository or many packages, not about how community repositories should work. Nothing here asks #18 or #23 to be amended. Both are true at different tiers, which is also how Debian works — the archive is versioned per package and resolved by a release; a third-party PPA is versioned as a unit and you take what it gives you. -Two practical notes for whoever reconciles these. First, adopting per-package versioning for the first-party archive costs #18 nothing: its meta-index, tap flow, and repository-level tags are unaffected, and the manifest reader this proposal needs is the same `TapIndex` cache [#23](https://github.com/cozystack/community/pull/23) already specifies. Second, #12's concrete surface — `tap` / `untap` / `init` / `push` / `search`, `community.`-prefixed source names, metadata-only index entries with an optional expected signing identity — is reusable as written; the package-level axis is being revived here on new grounds, not the specific ergonomics being re-litigated. +Two practical notes for whoever reconciles these. First, adopting per-package versioning for the first-party archive costs #18 nothing and asks nothing of it: its meta-index, tap flow, and repository-level tags are unaffected, and the manifest reader this proposal needs is the same `TapIndex` cache [#23](https://github.com/cozystack/community/pull/23) already specifies. Second, #12's concrete surface — `tap` / `untap` / `init` / `push` / `search`, `community.`-prefixed source names, metadata-only index entries with an optional expected signing identity — is reusable as written; the package-level axis is being revived here on new grounds, not the specific ergonomics being re-litigated. | Origin | Versioned unit | Rationale | |---|---|---| @@ -401,7 +470,7 @@ Either way, security fixes need an exception path: a patch release of the curren ## Testing -- **Conformance, in-tree and gating:** a synthetic package published at two versions; assert that upgrading the manifest from v1 to v2 reconciles exactly that package and leaves an untouched neighbour's `ExternalArtifact` digest and HelmRelease revision unchanged. This is the acceptance test for the entire proposal, and it subsumes the determinism check in rollout phase 2. +- **Conformance, in-tree and gating:** a synthetic package published at two versions; assert that upgrading the manifest from v1 to v2 reconciles exactly that package and leaves an untouched neighbour's `ExternalArtifact` digest and HelmRelease revision unchanged. This is the acceptance test for the entire proposal. - **CI gates:** content-changed-implies-bump, and schema-diff-classifies-bump, both run per PR over the package tree. - **Migration framework:** unit tests on the `cozy-lib` version helper including the string-vs-integer comparison at 10; per-package hook tests reusing the existing helm-unittest harness. - **Upgrade e2e:** a two-release chain (`2026.08` → `2026.09`) with a partial manifest diff, asserting untouched workloads are not restarted. The existing seed → upgrade → verify lane is the right host for this. @@ -410,24 +479,27 @@ Either way, security fixes need an exception path: a patch release of the curren ## Rollout -The ordering is chosen so that each phase is independently valuable and independently revertible, and so that the riskiest item (the versioned pool) is preceded by the experiment that de-risks it. +The ordering is chosen so that each phase is independently valuable and independently revertible, and so that the cheapest item with the largest measurable effect comes first. -1. **Cadence only.** Adopt the four-week train, publish the support window, cut `release-YYYY.MM` branches. No code changes. Two or three cycles of evidence before anything else depends on the cadence holding. -2. **Determinism check.** Confirm on kind (source-controller + source-watcher only) that a generated artifact's digest is stable for identical input content across two source revisions. Option A is already the chosen path per [§4](#4-the-versioned-pool); this phase only establishes whether it needs an upstream reproducibility fix alongside it. -3. **Manifest, inert.** Publish the release manifest with all versions equal to the current platform version. Add `version` to the data model. Nothing resolves differently. Add the two CI gates. -4. **ApplicationDefinition consolidation.** One coordinated pass folding this proposal's fields with #39's, #6's, and #3448's, before external catalogs depend on the shape. -5. **Migration framework.** `cozy-lib` migration helper; port the four existing ad-hoc hooks onto it; new migrations are authored per-package; the global lane is frozen except for cross-cutting cases. -6. **First real partial upgrade.** One release where the manifest moves a small number of low-risk packages (`bucket`, `http-cache` — not `postgres`, not `kubernetes`) and everything else holds. Measure what the tooling misses. -7. **`cozypkg`.** Versions, search, show, upgrade, hold, non-interactive output, and documentation. -8. **Repositories.** #43's catalogs and #18's taps land on the now-versioned mechanism; the manifest's `repositories` list ships the org catalog enabled-by-default per #43's shim plan. -9. **CalVer switch.** Rename the release stream once the manifest, partial upgrades, and the support window are all proven. Last, not first. +1. **Stop vendoring the tag.** Pin first-party images by digest alone in every chart, delete `hack/promote-rewrite-tags.sh` and its bats suite, and pass `--reproducible` to the pool push. No design commitment of any kind, and per [§4](#4-the-versioned-pool) it should stop 13 of the 35 packages a patch release moves. The effect is measurable on the next patch release, which makes it the honest test of whether the rest of this proposal is aimed at the right problem. +2. **Cadence only.** Adopt the four-week train, publish the support window, cut `release-YYYY.MM` branches. No code changes. Two or three cycles of evidence before anything else depends on the cadence holding. +3. **Read the source-watcher archive path.** Confirm that `ArtifactGenerator`'s re-tar normalises entry metadata the way the `flux` CLI's does; the CLI half is already established in [§4](#4-the-versioned-pool). This is a code read, not a cluster experiment. +4. **Extract core.** Move `cozystack-api`, `cozystack-controller` and `lineage-controller-webhook` out of the package pool into the bootstrap lane alongside the operator and the CRDs, and resolve the `packages/core/` naming collision (see [§1](#1-four-tiers-one-package-model)). This is the largest piece of work in the list and it is deliberately placed before any versioning change, because a tier boundary that delivery does not honour cannot carry a version. +5. **Manifest, inert.** Publish the release manifest with all versions equal to the current platform version. Add `version` to the data model. Nothing resolves differently. Add the two CI gates. +6. **ApplicationDefinition consolidation.** One coordinated pass folding this proposal's fields with #39's, #6's, and #3448's, before external catalogs depend on the shape. +7. **Migration framework.** `cozy-lib` migration helper; port the four existing ad-hoc hooks onto it; new migrations are authored per-package; the global lane is frozen except for cross-cutting cases. +8. **First real partial upgrade.** One release where the manifest moves a small number of low-risk packages (`bucket`, `http-cache` — not `postgres`, not `kubernetes`) and everything else holds. Measure what the tooling misses. +9. **`cozypkg`.** Versions, search, show, upgrade, hold, non-interactive output, and documentation. +10. **Repositories.** #43's catalogs and #18's taps land on the now-versioned mechanism; the manifest's `repositories` list ships the org catalog enabled-by-default per #43's shim plan. +11. **CalVer switch.** Rename the release stream once the manifest, partial upgrades, and the support window are all proven. Last, not first. ## Open questions -- **Is the generated artifact tarball reproducible?** Option A is chosen ([§4](#4-the-versioned-pool)), but if source-watcher's tarball carries unpack-time modification times, identical content can yield differing digests and partial upgrades need an upstream determinism fix alongside this work. Checkable on kind; the answer changes the implementation's dependencies, not the design. +- **Does source-watcher's re-tar normalise entry metadata?** The `flux` CLI's archive path does — mtime epoch, uid/gid zero, byte-identical output across builds ([§4](#4-the-versioned-pool)) — and the two share upstream archive code, but that has not been read. It is a code read rather than a cluster experiment, and the answer changes the implementation's dependencies, not the design. +- **Do `flux-plunger` and `flux-shard-operator` belong to core?** They exist only to make Flux deliver packages, which argues core; they are also Flux-version-coupled plumbing that a future delivery change would replace wholesale, which argues package. The classification in [§1](#1-four-tiers-one-package-model) puts them in core provisionally. - **Manifest kind and home.** A CRD applied to the cluster, a plain OCI artifact read by tooling, or both? #23's `TapIndex` cache is the obvious reader either way. - **How does the CalVer name relate to the existing `v1.x` stream?** Is `2026.08` a rename of the same stream, or does a final `v1.x` release announce the switch? What do existing release branches and backport automation do at the boundary? -- **Support window length.** Three releases? Six? Security-only tail? This must be answered before phase 1, not after. +- **Support window length.** Three releases? Six? Security-only tail? This must be answered before the cadence changes in rollout phase 2, not after. - **Does core ship between trains?** Recommendation above is no; maintainers should confirm. - **Version reporting and diagnostics.** With holds and partial upgrades, a cluster's state is a version vector rather than a single string. `cozypkg`, the dashboard, and the diagnostic bundle (`cozyreport` / crust-gather) must all carry it, or support gets harder rather than easier. Who owns that surface? - **Inter-package compatibility.** Debian works because packages have declared ABIs. Helm charts have none — the real interface is the values schema plus what one chart `lookup`s about another's live state, and several charts do exactly that (the Harbor jobservice storage-class preservation, the SeaweedFS fullname adoption). Should cross-package `lookup` be forbidden outside core, or should packages declare an interface version? @@ -437,10 +509,10 @@ The ordering is chosen so that each phase is independently valuable and independ ## Alternatives considered -- **Keep one version, improve the tooling.** Better changelogs and better test-impact analysis reduce the symptoms and leave the structure: every release still moves every chart, a breaking app change still forces a platform decision, and a package fix still cannot ship without a platform release. +- **Keep one version, improve the tooling.** Better changelogs and better test-impact analysis reduce the symptoms and leave the structure: a release still moves packages nothing changed in, a breaking app change still forces a platform decision, and a package fix still cannot ship without a platform release. Rollout phase 1 is the part of this alternative worth taking — it is cheap, it is real, and it is not a substitute for the rest. - **SemVer for the distribution too, no CalVer.** Rejected because the distribution's version has no honest semantic meaning once components carry their own — a monthly release containing one app major and forty patches is neither major nor minor. A date is truthful. It also removes the perverse incentive to avoid necessary breaking changes because "we are not ready for 2.0". - **CalVer everywhere, including packages.** Rejected: package consumers need compatibility information from the version, which is exactly what a date does not carry. -- **Repository-as-unit for everything (strict #18).** Adopted unchanged for community repositories, where its "tested together" reasoning holds. Not adopted for the first-party archive, for one reason: a repository-level version cannot express a partial upgrade, which is the proposal's primary goal. See [§8](#8-repository-as-unit-versus-package-as-unit) for why this is a narrow disagreement about the archive rather than a rejection of #18's model. +- **Repository-as-unit for everything (strict #18).** Adopted unchanged for community repositories, where its "tested together" reasoning holds. Not adopted for the first-party archive, for one reason: a repository-level version cannot express a partial upgrade, which is the proposal's primary goal. See [§8](#8-repository-as-unit-versus-package-as-unit) for why this extends #18 into its own Phase 2 rather than rejecting its model. - **Wait for the marketplace work and take per-package versioning as a side effect of it.** Rejected on sequencing. The internal costs enumerated in [Why this is a release-engineering proposal](#why-this-is-a-release-engineering-proposal-not-a-marketplace-proposal) are paid every cycle now and do not depend on any ecosystem work landing; tying their fix to a discovery surface that is still under discussion delays it for reasons unrelated to it. The dependency runs the other way — the marketplace benefits from a versioned archive, not the reverse. - **Per-package versions but no manifest — resolve with SemVer ranges at install time.** Rejected firmly. Ranges without a lockfile import npm's resolution problem into a platform with no lockfile and no ability to test the resolved set. The manifest *is* the lockfile. - **Let users compose versions freely.** Rejected as a supported mode. A distribution decides the set; that is what makes it testable and supportable. Holds exist as an escape hatch and are labelled unsupported. @@ -449,12 +521,12 @@ The ordering is chosen so that each phase is independently valuable and independ ## Appendix A — measured facts -Collected from `main` on 2026-07-27 (post-`v1.6.0`). Line references are to that state. +Collected from `main` on 2026-07-27 (post-`v1.6.0`) and re-measured on 2026-08-27 (post-`v1.6.2`). Line references are to the later state. | Fact | Value | Source | |---|---|---| -| Charts on the `0.0.0` build-time placeholder | 160 of 164 (excluding vendored `charts/`) | `packages/**/Chart.yaml` | -| Charts already carrying a hand-maintained version | 4 — `apps/foundationdb` `0.1.0`, `system/cozystack-scheduler` `0.3.0`, `system/kubeovn` `0.38.0`, `tests/cozy-lib-tests` `0.1.0`. Nothing reads them; the inconsistency is itself evidence that the convention is unenforced | same | +| Charts on the `0.0.0` placeholder | 164 of 164 at depth 2 (excluding vendored `charts/`) | `packages/*/*/Chart.yaml` | +| Anything that replaces that placeholder in the pool | nothing. `helm package --version` runs only in the `repo` target, which builds `_out/repos/`; `fix-charts` resets the field to `0.0.0` | `packages/{apps,system,extra,library}/Makefile` | | Package directories | 23 apps, 126 system, 8 extra | `packages/` | | `PackageSource` objects shipped | 98, all referencing `cozystack-packages` | `packages/core/platform/sources/*.yaml` | | `OCIRepository` objects in play | 2 (`cozystack-platform`, its clone `cozystack-packages`) | `main.go:564-602`, `repository.yaml:19` | @@ -465,7 +537,12 @@ Collected from `main` on 2026-07-27 (post-`v1.6.0`). Line references are to that | `cozypkg` size / commits / documentation | 1811 lines / 3 commits / 0 lines of docs | `cmd/cozypkg/`, `git log` | | Core code churn, 12 months | ~620 commits across `internal/`, `pkg/`, `api/` | `git log --since='12 months ago'` | | Generated artifact revision derivation | content digest — the field is optional and cozystack never sets it | `fluxcd.yaml:522-528` (CRD doc), `packagesource_reconciler.go:241-244` (no `Revision`) | -| Consequence | the delivery layer would already skip unchanged packages; the per-release chart-version rewrite is what guarantees none are ever unchanged | `packages/core/installer/Makefile` | +| Packages moved by a patch release | 35 of 164 — 9 chart source, 13 image digest, **13 tag string only** — 129 untouched | `git diff v1.6.1..v1.6.2 -- packages/` | +| Packages moved by a minor release | 92 of 164 | `git diff v1.5.4..v1.6.0 -- packages/` | +| Charts building from the root Go context | 11 (`COPY api pkg cmd internal`), so one `internal/` change moves every one of their digests | `packages/*/*/Makefile`, `images/*/Dockerfile` | +| Flux archive determinism | byte-identical across builds; entries normalised to mtime `1970-01-01`, uid/gid `0` | `flux build artifact` twice on one tree, flux 2.8.6 | +| `--reproducible` on the pool push | available, not passed | `packages/core/installer/Makefile:37` | +| Consequence | the delivery layer already skips unchanged packages; what defeats it is the version substring vendored into image references and rewritten at promotion | `hack/promote-rewrite-tags.sh`, `hack/lib/image-refs.sh` | ## Appendix B — unrelated defects found while surveying From 823ad450ea87a7954fcea231dc407e1d01cc3f27 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Thu, 27 Aug 2026 17:15:37 +0500 Subject: [PATCH 05/13] docs(cozystack-as-a-distribution): make the manifest the unit of testing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The strongest objection to per-package versioning is that testing cost explodes into an N-by-M matrix, and #43 names that cost as its own driver. Answer it where the answer already is: the release manifest is a tested-together set by definition, e2e certifies the pinned combination, and per-package versions give the manifest something to pin rather than multiplying what must be run. A package that ships between trains is bounded by a declared requiresCore range exercised at its endpoints — Kubernetes' version skew policy, one tier down — instead of by re-testing everything. Both precedents carry a piece that is missing here, so the section states three preconditions rather than claiming the model is free. cozystack#3276 already builds the right harness: install previous stable, seed workloads with canary data, upgrade, verify survival, data integrity, all-HelmReleases-Ready, PVs Bound and the migration stamp advanced. It is opt-in by label and gates nothing, which is right for a lane nothing depends on and wrong for the lane that certifies a release. Recorded as a prerequisite and added to the rollout, since everything past the manifest needs to be able to test an upgrade of one. The earlier text called this lane "existing", which it is not yet. A declared range that no job exercises is documentation. Kubernetes publishes its skew policy and also runs skew jobs; the equivalent here is two endpoint runs per out-of-train package, not a matrix. The third precondition is the one with no precedent-supplied answer, and it was already an open question here: package-to-package edges are invisible. DependsOn is a list of names with no version constraint — 52 edges that order installation and say nothing about compatibility — and the larger coupling is undeclared entirely, with 35 charts reaching into each other's live state across 267 lookup sites, several reading another package's CRDs. Such a break changes no values schema and involves no core version, so neither gate sees it; the lookup returns empty and the template renders something else. The open question is promoted to a prerequisite and the guarantee is described as covering the edges the manifest can see. The acceptance test gains its missing half: #3276 asserts that every HelmRelease reconciled, and a partial upgrade needs the complement — that those outside the manifest diff did not. Assisted-By: Claude Signed-off-by: Myasnikov Daniil --- .../cozystack-as-a-distribution/README.md | 55 ++++++++++++++----- 1 file changed, 40 insertions(+), 15 deletions(-) diff --git a/design-proposals/cozystack-as-a-distribution/README.md b/design-proposals/cozystack-as-a-distribution/README.md index 7611284..33a763b 100644 --- a/design-proposals/cozystack-as-a-distribution/README.md +++ b/design-proposals/cozystack-as-a-distribution/README.md @@ -30,6 +30,7 @@ This proposal is deliberately a joining piece. Several accepted or in-flight pro | [community#6](https://github.com/cozystack/community/pull/6) — ApplicationDefinition multi-version conversion (`@kvaps`, Draft) | `versions[]` with a storage version, `to`/`from` conversion templates, a `_version` stamp, a background migration controller | **Orthogonal axis, and the thing that makes semver honest.** #6 versions the *API surface* of an app; this proposal versions the *package*. Their relationship is what gives "breaking change" a testable definition — see [What MAJOR means](#what-major-means) | | [community#39](https://github.com/cozystack/community/pull/39) — Fold `extra` into `apps` (`@myasnikovdaniil`, Draft) | Retires the `extra` bucket; visibility / cardinality / protection / capability as declarative `ApplicationDefinition` fields | **Same direction, one tier down.** #39 removes a directory-as-metadata convention by moving the metadata onto the ApplicationDefinition; this proposal does the same for the version. Together they turn ApplicationDefinition into the package's control file | | [cozystack#3448](https://github.com/cozystack/cozystack/pull/3448) — ApplicationGroupDefinition (MVP) | Dynamic API-group registration, reserved-namespace validation, restart-based pickup via the `cozystack.io/config-hash` rollout | **Prerequisite for tier 2, and the proof that the runtime is already data-driven.** The mechanism it extends is the reason per-package delivery is feasible at all | +| [cozystack#3276](https://github.com/cozystack/cozystack/pull/3276) — chainsaw-native release upgrade lane (`@myasnikovdaniil`, open) | Install previous stable → seed workloads with canary data → upgrade → verify survival, data integrity, all-HelmReleases-Ready, PVs Bound, migration stamp advanced | **Prerequisite, and the thing that makes "tested together" a fact.** Under this proposal the manifest is the unit of testing, so the lane that exercises an upgrade of the manifest stops being advisory — see [Testing](#testing) | | [community#21](https://github.com/cozystack/community/pull/21) — Self-hosted in-cluster registry for air-gap (`@gecube`, Draft) | Offline bundle, in-cluster registry as source of truth | **Affected, and improved.** A per-package pool changes what an offline bundle contains; a release manifest is exactly the mirror list an air-gapped site needs. Flagged as an open question rather than solved here | | [community#25](https://github.com/cozystack/community/pull/25) (Draft), [community#33](https://github.com/cozystack/community/pull/33) (**accepted**, merged 2026-07-18) | Per-cluster etcd; ComputePlane as an operator-owned module | Downstream consumers of the tiering; not blocked by this proposal | @@ -90,7 +91,7 @@ One counter-fact to handle: `api/apps/v1alpha1/` is a **separate Go module** (`a - **A release moves packages that did not change, and a decorative tag string is why.** Measured across `v1.6.1..v1.6.2`, 35 of the 164 packages have any changed file at all: 9 changed chart source, 13 had an image digest move, and **13 moved because the version substring inside an image reference was rewritten while the digest stayed byte-identical.** The remaining 129 are untouched. Since generated artifact revisions are content-derived (see [§4](#4-the-versioned-pool)), each of those 13 nevertheless gets a new artifact digest and a HelmRelease upgrade, for a change Kubernetes never reads — it pulls by digest. So the churn is neither total nor unavoidable; it is manufactured by one line of cosmetics per chart, and removing it is a smaller change than anything else in this proposal. - **Breaking changes are held hostage.** A breaking change in one app forces either a platform major (which frightens users away from an upgrade that is mostly bug fixes) or an indefinite delay. There is no way to say "MongoDB 4.0 is breaking, the platform is not". - **A package fix cannot ship without a platform release.** Today the only route to a released cluster is a backport, and the backport bot fails silently on conflict and imports whole files for files absent on the target branch. Security fixes inherit that latency. -- **Nothing can be tested or supported at package granularity,** because nothing *is* a package at release granularity. #43 wants catalog repositories with their own CI against a matrix of supported Cozystack releases; there is no version to put in that matrix except the whole platform's. +- **Nothing can be tested or supported at package granularity,** because nothing *is* a package at release granularity. #43 wants catalog repositories with their own CI against a matrix of supported Cozystack releases; there is no version to put in that matrix except the whole platform's. The answer is not a matrix — it is a manifest that names the tested set, plus a declared range for anything that leaves it; see [Testing](#testing). - **The word "version" has no defined meaning.** Nobody has had to decide what makes a chart change breaking, because no chart version has ever been read by anything. ### Why this is a release-engineering proposal, not a marketplace proposal @@ -268,7 +269,7 @@ Two things follow. First, "every release moves every chart" was never true: a pa An earlier draft listed as an open question whether the generated tarball is reproducible for identical input content, on the theory that unpack-time modification times could vary the digest. **It does not.** Flux normalises the archive: entries are written with mtime `1970-01-01`, uid/gid `0`, and fixed modes. Building the same directory twice with `flux build artifact` — pinned at 2.8.6, the version all three release workflows install — produces byte-identical output, `touch` in between included. -One residual, stated precisely rather than waved away: that exercises the `flux` CLI's archive path, not source-watcher's re-tar inside `ArtifactGenerator`. The two share the upstream `fluxcd/pkg` archive code, so the remaining check is reading that code path rather than standing up a cluster, and rollout phase 3 shrinks accordingly. +One residual, stated precisely rather than waved away: that exercises the `flux` CLI's archive path, not source-watcher's re-tar inside `ArtifactGenerator`. The two share the upstream `fluxcd/pkg` archive code, so the remaining check is reading that code path rather than standing up a cluster, and rollout phase 4 shrinks accordingly. A related loose end worth fixing in the same pass: `flux push artifact --reproducible` exists — it fixes the OCI created-timestamp at epoch — and `packages/core/installer/Makefile:37` does not pass it. That affects the pool manifest's own digest, not the per-component artifacts, so it does not cause the churn above; it is simply free determinism that is currently declined. @@ -470,10 +471,33 @@ Either way, security fixes need an exception path: a patch release of the curren ## Testing -- **Conformance, in-tree and gating:** a synthetic package published at two versions; assert that upgrading the manifest from v1 to v2 reconciles exactly that package and leaves an untouched neighbour's `ExternalArtifact` digest and HelmRelease revision unchanged. This is the acceptance test for the entire proposal. -- **CI gates:** content-changed-implies-bump, and schema-diff-classifies-bump, both run per PR over the package tree. +The obvious reading of per-package versioning is that testing cost explodes: N packages against M core versions, and nobody can afford the matrix. #43 names that cost as its central driver, and it is the strongest practical objection to this proposal. + +**The manifest is the answer, and it is the reason the manifest exists.** A release manifest is a tested-together set by definition — e2e runs against the pinned combination, and what it certifies is that combination, not each package in isolation. There is one combination per release, so the cost is what it is today. Per-package versions do not multiply the matrix; they give the manifest something to pin. + +A package that ships between trains is the only case that leaves the certified set, and it is bounded by declaration rather than by re-testing everything: the package carries a `requiresCore` range and is exercised at that range's endpoints. That is the shape of Kubernetes' [version skew policy](https://kubernetes.io/releases/version-skew-policy/) — kubelet may trail the apiserver by three minors, and the project supports that because the skew is declared and exercised, not because every pair is tried. It is also the shape of a distribution release: Debian's `Release` file pins a set, and a package migrates into it only after its own tests and the tests of its reverse-dependencies pass. + +### Three preconditions + +Both precedents carry a piece Cozystack does not have yet, and the model is only as honest as these. + +1. **The upgrade lane must land, and must stop being advisory.** [cozystack#3276](https://github.com/cozystack/cozystack/pull/3276) builds exactly the right harness — install the previous stable, seed real workloads with canary data, upgrade, then verify survival, data integrity, all-HelmReleases-Ready, PVs still Bound and the migration stamp advanced. It is currently opt-in by label and gates nothing, which is the correct setting for a lane nobody depends on and the wrong one for the lane that certifies a release. Under this proposal it becomes the mechanism that makes "tested together" a fact rather than a claim, so landing it is a prerequisite rather than an adjacent improvement. Its findings already argue for it: the lane is red on upgrade-only defects that no other suite reaches. + +2. **A declared range must be exercised at its endpoints.** Kubernetes does not merely publish its skew policy; it runs skew jobs. A package released between trains with `requiresCore: ">=1.7.0 <2.0.0"` is tested against `1.7.0` and against the newest `1.x` in the manifest — two points, not a matrix. A range that no job ever exercises is documentation, and the first upgrade that violates it will be discovered by an operator. + +3. **Package-to-package edges must become visible, or the guarantee has holes it cannot see.** This is the gap that has no precedent-supplied answer, and it is [an open question below](#open-questions) promoted to a prerequisite. Debian can compute which reverse-dependencies to re-test because `Depends: libfoo (>= 1.2)` is a versioned edge. Cozystack's equivalent is `DependsOn []string` (`api/v1alpha1/packagesource_types.go:74`, and the per-component form at `:135`) — 52 declared edges carrying names and no constraints, which orders installation and says nothing about compatibility. + + The larger part is not declared at all. Charts read each other's live state: of 267 `lookup` call sites across 35 charts, most are the ordinary self-referential idiom, but several reach into another package's CRDs — `postgresql.cnpg.io/v1` `Cluster` (5), `instancetype.kubevirt.io/v1beta1` `VirtualMachineClusterInstancetype` (5), `cozystack.io/v1alpha1` `Package` (2), plus `MachineSet`, `DataVolume`, `BucketClaim` and `StorageClass`. A chart's *rendering* therefore depends on a CRD version another package installs. `requiresCore` cannot express that — the dependency is not on core. A values-schema diff cannot detect a break in it — the schema did not change. The `lookup` simply returns empty and the template renders something else, silently. + + Either cross-package `lookup` is forbidden outside core, or a package declares the interfaces it consumes with a version. Until one of those lands, the manifest's guarantee covers the edges it can see, and the document should say so rather than imply completeness. + +### The tests themselves + +- **The acceptance test for the whole proposal, and the one negative assertion nothing makes today:** a manifest diff that moves one package must reconcile that package and leave an untouched neighbour's `ExternalArtifact` digest *and* HelmRelease revision unchanged. #3276 already asserts that every HelmRelease reconciled; partial upgrades need the complement — that the ones outside the diff did not. A synthetic package published at two versions is enough to pin it in-tree. +- **Upgrade e2e over a two-release chain** (`2026.08` → `2026.09`) with a partial manifest diff, asserting untouched workloads are not restarted. Hosted by #3276's lane once it lands. +- **Skew jobs at the declared endpoints,** per precondition 2, for any package released outside a train. +- **CI gates:** content-moved-implies-version-moved, and schema-diff-classifies-the-declared-bump — the second reusing `cmd/api-gate`'s verdict shape, which already computes "sizeable or breaking" for the API surface (see [§5](#5-what-major-means)). - **Migration framework:** unit tests on the `cozy-lib` version helper including the string-vs-integer comparison at 10; per-package hook tests reusing the existing helm-unittest harness. -- **Upgrade e2e:** a two-release chain (`2026.08` → `2026.09`) with a partial manifest diff, asserting untouched workloads are not restarted. The existing seed → upgrade → verify lane is the right host for this. - **Manifest validation:** every package in a manifest resolves; `requiresCore` is satisfiable; `supportedUpgradeFrom` chains are acyclic and reachable. - **Load:** if Option B is chosen, source-controller and source-watcher behaviour with ~100 `OCIRepository` objects, including source-watcher pod restart with a cold `emptyDir`. @@ -483,15 +507,16 @@ The ordering is chosen so that each phase is independently valuable and independ 1. **Stop vendoring the tag.** Pin first-party images by digest alone in every chart, delete `hack/promote-rewrite-tags.sh` and its bats suite, and pass `--reproducible` to the pool push. No design commitment of any kind, and per [§4](#4-the-versioned-pool) it should stop 13 of the 35 packages a patch release moves. The effect is measurable on the next patch release, which makes it the honest test of whether the rest of this proposal is aimed at the right problem. 2. **Cadence only.** Adopt the four-week train, publish the support window, cut `release-YYYY.MM` branches. No code changes. Two or three cycles of evidence before anything else depends on the cadence holding. -3. **Read the source-watcher archive path.** Confirm that `ArtifactGenerator`'s re-tar normalises entry metadata the way the `flux` CLI's does; the CLI half is already established in [§4](#4-the-versioned-pool). This is a code read, not a cluster experiment. -4. **Extract core.** Move `cozystack-api`, `cozystack-controller` and `lineage-controller-webhook` out of the package pool into the bootstrap lane alongside the operator and the CRDs, and resolve the `packages/core/` naming collision (see [§1](#1-four-tiers-one-package-model)). This is the largest piece of work in the list and it is deliberately placed before any versioning change, because a tier boundary that delivery does not honour cannot carry a version. -5. **Manifest, inert.** Publish the release manifest with all versions equal to the current platform version. Add `version` to the data model. Nothing resolves differently. Add the two CI gates. -6. **ApplicationDefinition consolidation.** One coordinated pass folding this proposal's fields with #39's, #6's, and #3448's, before external catalogs depend on the shape. -7. **Migration framework.** `cozy-lib` migration helper; port the four existing ad-hoc hooks onto it; new migrations are authored per-package; the global lane is frozen except for cross-cutting cases. -8. **First real partial upgrade.** One release where the manifest moves a small number of low-risk packages (`bucket`, `http-cache` — not `postgres`, not `kubernetes`) and everything else holds. Measure what the tooling misses. -9. **`cozypkg`.** Versions, search, show, upgrade, hold, non-interactive output, and documentation. -10. **Repositories.** #43's catalogs and #18's taps land on the now-versioned mechanism; the manifest's `repositories` list ships the org catalog enabled-by-default per #43's shim plan. -11. **CalVer switch.** Rename the release stream once the manifest, partial upgrades, and the support window are all proven. Last, not first. +3. **Land the upgrade lane.** Get [cozystack#3276](https://github.com/cozystack/cozystack/pull/3276) merged and promote it from advisory to required for release PRs. Independently valuable — it is already finding upgrade-only defects nothing else reaches — and everything downstream of the manifest depends on being able to test an upgrade of one. +4. **Read the source-watcher archive path.** Confirm that `ArtifactGenerator`'s re-tar normalises entry metadata the way the `flux` CLI's does; the CLI half is already established in [§4](#4-the-versioned-pool). This is a code read, not a cluster experiment. +5. **Extract core.** Move `cozystack-api`, `cozystack-controller` and `lineage-controller-webhook` out of the package pool into the bootstrap lane alongside the operator and the CRDs, and resolve the `packages/core/` naming collision (see [§1](#1-four-tiers-one-package-model)). This is the largest piece of work in the list and it is deliberately placed before any versioning change, because a tier boundary that delivery does not honour cannot carry a version. +6. **Manifest, inert.** Publish the release manifest with all versions equal to the current platform version. Add `version` to the data model. Nothing resolves differently. Add the two CI gates. +7. **ApplicationDefinition consolidation.** One coordinated pass folding this proposal's fields with #39's, #6's, and #3448's, before external catalogs depend on the shape. +8. **Migration framework.** `cozy-lib` migration helper; port the four existing ad-hoc hooks onto it; new migrations are authored per-package; the global lane is frozen except for cross-cutting cases. +9. **First real partial upgrade.** One release where the manifest moves a small number of low-risk packages (`bucket`, `http-cache` — not `postgres`, not `kubernetes`) and everything else holds. Measure what the tooling misses. +10. **`cozypkg`.** Versions, search, show, upgrade, hold, non-interactive output, and documentation. +11. **Repositories.** #43's catalogs and #18's taps land on the now-versioned mechanism; the manifest's `repositories` list ships the org catalog enabled-by-default per #43's shim plan. +12. **CalVer switch.** Rename the release stream once the manifest, partial upgrades, and the support window are all proven. Last, not first. ## Open questions @@ -502,7 +527,7 @@ The ordering is chosen so that each phase is independently valuable and independ - **Support window length.** Three releases? Six? Security-only tail? This must be answered before the cadence changes in rollout phase 2, not after. - **Does core ship between trains?** Recommendation above is no; maintainers should confirm. - **Version reporting and diagnostics.** With holds and partial upgrades, a cluster's state is a version vector rather than a single string. `cozypkg`, the dashboard, and the diagnostic bundle (`cozyreport` / crust-gather) must all carry it, or support gets harder rather than easier. Who owns that surface? -- **Inter-package compatibility.** Debian works because packages have declared ABIs. Helm charts have none — the real interface is the values schema plus what one chart `lookup`s about another's live state, and several charts do exactly that (the Harbor jobservice storage-class preservation, the SeaweedFS fullname adoption). Should cross-package `lookup` be forbidden outside core, or should packages declare an interface version? +- **Inter-package compatibility — promoted to a prerequisite, still unanswered.** Debian works because packages have declared ABIs. Helm charts have none: `DependsOn` is a list of names with no version, and the real interface is the values schema plus what one chart `lookup`s about another's live state — which 35 charts do across 267 call sites, several of them reaching into another package's CRDs. Should cross-package `lookup` be forbidden outside core, or should packages declare the interfaces they consume with a version? [Testing](#testing) explains why the manifest's guarantee is incomplete until this is settled; it does not settle it. - **Air-gap.** #21's bundle currently mirrors one artifact. With a per-package pool, what does the bundle contain, and does the manifest become the mirror spec? Coordinate with #21 rather than deciding here. - **The `api/apps/v1alpha1` Go module** is tagged in lockstep with the platform and mirrors app schemas for external consumers. Per-app modules, or an explicit statement that the module tracks core rather than apps? - **Naming.** "Variant" already means two different things — the installer's `talos|generic|hosted` and `PackageSource.spec.variants[]`. Adding "version", "section", "repository", and "channel" to the same vocabulary needs a glossary, or reviews will go sideways. From d7dd31ae265f06469c752e8d3499e93c68c92172 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Thu, 27 Aug 2026 18:16:03 +0500 Subject: [PATCH 06/13] docs(cozystack-as-a-distribution): replace the estimate with a cluster measurement MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The churn figures in section 4 were derived from `git diff` and predicted 35 of 164 packages moving on v1.6.1 -> v1.6.2. The prediction was run: a three-node Talos stand, cozy-installer 1.6.1 with the isp-full variant covering 157 of the 164 package directories as 207 ExternalArtifacts, a snapshot of every artifact digest and HelmRelease revision, helm upgrade to 1.6.2, and the same snapshot once the platform re-converged at 95/95 Ready. Result, with every moved artifact attributed and none left over: 134 held, 18 moved for the tag string alone, 25 for a cozy-lib fan-out, 18 for a genuine image rebuild, 12 for changed chart source. 23 of 95 HelmReleases took a new revision and 43 of 164 pods were replaced. Two things the diff could not show. The cosmetic case restarted the data plane: Cilium, LINSTOR, MetalLB and objectstorage-controller are all tag-only movers, and they restarted 3, 8, 4 and 1 pods; the metallb pods came back running the same two digests packages/system/metallb/values.yaml carries at both tags. And the ArtifactGenerator copies library charts into each consuming package's artifact — 36 artifacts vendor cozy-lib — so one edit to _barman.tpl moved 25 otherwise-untouched packages. A package directory is not self-contained, which is why the diff-based prediction missed them. That second cause is not removable the way the tag is: the library really is part of the rendered chart. It means a package's version is a function of its directory and its vendored libraries, and it makes cozy-lib an unversioned part of 36 packages' interface — the undeclared-interface problem Testing raises for cross-package lookup, showing up inside the first-party archive, and the tractable half of it since the edge is visible in the copy operations. The run also settles the last open question by observation rather than by reading upstream code: 134 artifacts held a byte-identical digest across two different pool revisions, which is only possible if source-watcher's re-tar normalises entry metadata as the flux CLI's does. Assisted-By: Claude Signed-off-by: Myasnikov Daniil --- .../cozystack-as-a-distribution/README.md | 67 +++++++++++++++---- 1 file changed, 53 insertions(+), 14 deletions(-) diff --git a/design-proposals/cozystack-as-a-distribution/README.md b/design-proposals/cozystack-as-a-distribution/README.md index 33a763b..eaa3322 100644 --- a/design-proposals/cozystack-as-a-distribution/README.md +++ b/design-proposals/cozystack-as-a-distribution/README.md @@ -252,24 +252,55 @@ packages/system/metallb/values.yaml + tag: v1.6.2@sha256:9d8ba76cdb9c7c6221334ad05d706dee22b138b3e90c1fe8fc884925b7480c02 ``` -Identical digest, different file. Kubernetes resolves this reference by digest and never reads the tag, but the chart's bytes changed, so its generated artifact digest changed, so its HelmRelease upgraded. Classified across the whole tree for that release: +Identical digest, different file. Kubernetes resolves this reference by digest and never reads the tag, but the chart's bytes changed, so its generated artifact digest changed, so its HelmRelease upgraded. -| `v1.6.1` → `v1.6.2` | packages | -|---|---| -| untouched | **129** | -| tag string only, image digest unchanged | **13** | -| image digest moved (rebuild) | 13 | -| chart source changed | 9 | +#### Measured on a cluster, not inferred from the diff + +The above was worked out from `git diff`, which predicts 35 of 164 packages moving on that release. That prediction was then run: a three-node Talos stand, `cozy-installer` 1.6.1 with the `isp-full` variant (207 `ExternalArtifact` objects covering 157 of the 164 package directories), a baseline snapshot of every artifact digest and HelmRelease revision, `helm upgrade` to 1.6.2, and the same snapshot again once the platform re-converged at 95/95 HelmReleases Ready. Every artifact that moved was then attributed to a cause, with none left over: + +| `v1.6.1` → `v1.6.2`, 207 artifacts | count | cause | +|---|---|---| +| **held** — digest unchanged | **134** | nothing in the package or its libraries changed | +| moved | 18 | **tag string only — the image is byte-identical** | +| moved | 25 | **`cozy-lib` fan-out** (see below) | +| moved | 18 | image digest genuinely moved (rebuild) | +| moved | 12 | chart source changed | + +Downstream: 23 of 95 HelmReleases took a new revision, and **43 of 164 pods were replaced.** + +Three conclusions, and the first two are the proposal's case. + +**"Every release moves every chart" was never true.** 134 of 207 artifacts held. The delivery layer already skips what does not change; the question was only ever what makes things change. + +**Of the 73 artifacts that moved, 43 moved for a reason that has nothing to do with the package.** 18 for a tag string, 25 for a library fan-out. Only 30 moved because something inside the package itself changed. + +**The cosmetic case is not academic — it restarted the data plane.** Among the tag-only movers are `system/cilium` (in five variants), `system/linstor`, `system/metallb` and `system/objectstorage-controller`, and on this cluster they restarted 3, 8, 4 and 1 pods respectively. The metallb pods came back running `sha256:9d8ba76c…` and `sha256:87df3c82…` — the exact digests `packages/system/metallb/values.yaml` carries at **both** tags. The CNI and the storage layer were restarted to deliver a string no runtime reads. -Two things follow. First, "every release moves every chart" was never true: a patch release moves 35 of 164, and a minor (`v1.5.4` → `v1.6.0`) moves 92. Second, 13 of those 35 move for a string no runtime reads. +#### The second cause: a library is vendored into every consumer -**The fix is smaller than any other item in this proposal and independent of all of them: stop vendoring the tag.** Write `ghcr.io/cozystack/cozystack/metallb@sha256:…` and nothing else. The tag continues to exist in the registry — `hack/promote-retag.sh` pushes it — and the release manifest names it, so nothing that a human or a mirror needs is lost; only the copy that sits inside the chart and forces a reconcile goes away. `hack/promote-rewrite-tags.sh` and its bats suite are deleted outright, and the rc-leftover scan in `hack/verify-promoted-packages.sh` has nothing left to scan for. This is worth doing on its own merits whatever happens to the rest of the proposal, and its effect is measurable on the very next patch release. +The 25 unattributed movers were the finding the `git diff` could not have produced, because it assumes a package directory is self-contained. It is not. The operator's `ArtifactGenerator` copies library charts into each consuming package's artifact — for `apps/redis`, verbatim from the cluster: + +```yaml +copy: +- from: '@cozystack-packages/apps/redis/**' + to: '@artifact/redis/' +- from: '@cozystack-packages/library/cozy-lib/**' + to: '@artifact/redis/charts/cozy-lib/' +``` + +36 of the 207 artifacts bundle `cozy-lib` this way, and `packages/library/cozy-lib/templates/_barman.tpl` changed in this release. One edit to one library template therefore moved 25 packages, every one of which is otherwise untouched. + +This is not cosmetic and cannot be deleted the way the tag can — the library genuinely is part of the rendered chart. It is a statement about what a package *is*: a package's version must be a function of its own directory **and** the libraries it vendors, and a `cozy-lib` change legitimately bumps every consumer. Which makes `cozy-lib` a de-facto part of 36 packages' interface with no version on it — the same undeclared-interface problem [Testing](#testing) raises for cross-package `lookup`, appearing inside the first-party archive rather than between catalogs. Partial upgrades will not isolate a `cozy-lib` fix, and the design should say so rather than discover it later. + +#### Stop vendoring the tag + +**The tag fix is smaller than any other item in this proposal and independent of all of them.** Write `ghcr.io/cozystack/cozystack/metallb@sha256:…` and nothing else. The tag continues to exist in the registry — `hack/promote-retag.sh` pushes it — and the release manifest names it, so nothing that a human or a mirror needs is lost; only the copy that sits inside the chart and forces a reconcile goes away. `hack/promote-rewrite-tags.sh` and its bats suite are deleted outright, and the rc-leftover scan in `hack/verify-promoted-packages.sh` has nothing left to scan for. This is worth doing on its own merits whatever happens to the rest of the proposal, and its effect is measurable on the very next patch release. #### The tarball determinism question is answered An earlier draft listed as an open question whether the generated tarball is reproducible for identical input content, on the theory that unpack-time modification times could vary the digest. **It does not.** Flux normalises the archive: entries are written with mtime `1970-01-01`, uid/gid `0`, and fixed modes. Building the same directory twice with `flux build artifact` — pinned at 2.8.6, the version all three release workflows install — produces byte-identical output, `touch` in between included. -One residual, stated precisely rather than waved away: that exercises the `flux` CLI's archive path, not source-watcher's re-tar inside `ArtifactGenerator`. The two share the upstream `fluxcd/pkg` archive code, so the remaining check is reading that code path rather than standing up a cluster, and rollout phase 4 shrinks accordingly. +**The cluster run settles the residual too.** That local check exercises the `flux` CLI's archive path rather than source-watcher's re-tar inside `ArtifactGenerator`, which left a gap. The upgrade above closes it by observation: 134 artifacts held a byte-identical digest across two *different* pool revisions, which is only possible if source-watcher's re-tar normalises entry metadata exactly as the CLI's does. No code read and no kind cluster are needed; the question is answered. A related loose end worth fixing in the same pass: `flux push artifact --reproducible` exists — it fixes the OCI created-timestamp at epoch — and `packages/core/installer/Makefile:37` does not pass it. That affects the pool manifest's own digest, not the per-component artifacts, so it does not cause the churn above; it is simply free determinism that is currently declined. @@ -489,6 +520,8 @@ Both precedents carry a piece Cozystack does not have yet, and the model is only The larger part is not declared at all. Charts read each other's live state: of 267 `lookup` call sites across 35 charts, most are the ordinary self-referential idiom, but several reach into another package's CRDs — `postgresql.cnpg.io/v1` `Cluster` (5), `instancetype.kubevirt.io/v1beta1` `VirtualMachineClusterInstancetype` (5), `cozystack.io/v1alpha1` `Package` (2), plus `MachineSet`, `DataVolume`, `BucketClaim` and `StorageClass`. A chart's *rendering* therefore depends on a CRD version another package installs. `requiresCore` cannot express that — the dependency is not on core. A values-schema diff cannot detect a break in it — the schema did not change. The `lookup` simply returns empty and the template renders something else, silently. + The first-party archive already has an instance of this, measured rather than hypothesised: `cozy-lib` is copied into 36 artifacts, and one edit to it moved 25 packages on a real upgrade ([§4](#4-the-versioned-pool)). That edge is at least *visible* in the `ArtifactGenerator`'s copy operations, which the `lookup` edges are not — so it is the tractable half of the same problem and the natural place to start. + Either cross-package `lookup` is forbidden outside core, or a package declares the interfaces it consumes with a version. Until one of those lands, the manifest's guarantee covers the edges it can see, and the document should say so rather than imply completeness. ### The tests themselves @@ -505,7 +538,7 @@ Both precedents carry a piece Cozystack does not have yet, and the model is only The ordering is chosen so that each phase is independently valuable and independently revertible, and so that the cheapest item with the largest measurable effect comes first. -1. **Stop vendoring the tag.** Pin first-party images by digest alone in every chart, delete `hack/promote-rewrite-tags.sh` and its bats suite, and pass `--reproducible` to the pool push. No design commitment of any kind, and per [§4](#4-the-versioned-pool) it should stop 13 of the 35 packages a patch release moves. The effect is measurable on the next patch release, which makes it the honest test of whether the rest of this proposal is aimed at the right problem. +1. **Stop vendoring the tag.** Pin first-party images by digest alone in every chart, delete `hack/promote-rewrite-tags.sh` and its bats suite, and pass `--reproducible` to the pool push. No design commitment of any kind. Measured on a cluster ([§4](#4-the-versioned-pool)), this removes 18 of the 73 artifact moves a patch release causes, and with them the Cilium, LINSTOR, MetalLB and objectstorage-controller pod restarts that deliver no new bytes. 2. **Cadence only.** Adopt the four-week train, publish the support window, cut `release-YYYY.MM` branches. No code changes. Two or three cycles of evidence before anything else depends on the cadence holding. 3. **Land the upgrade lane.** Get [cozystack#3276](https://github.com/cozystack/cozystack/pull/3276) merged and promote it from advisory to required for release PRs. Independently valuable — it is already finding upgrade-only defects nothing else reaches — and everything downstream of the manifest depends on being able to test an upgrade of one. 4. **Read the source-watcher archive path.** Confirm that `ArtifactGenerator`'s re-tar normalises entry metadata the way the `flux` CLI's does; the CLI half is already established in [§4](#4-the-versioned-pool). This is a code read, not a cluster experiment. @@ -520,7 +553,8 @@ The ordering is chosen so that each phase is independently valuable and independ ## Open questions -- **Does source-watcher's re-tar normalise entry metadata?** The `flux` CLI's archive path does — mtime epoch, uid/gid zero, byte-identical output across builds ([§4](#4-the-versioned-pool)) — and the two share upstream archive code, but that has not been read. It is a code read rather than a cluster experiment, and the answer changes the implementation's dependencies, not the design. +- **~~Is the generated tarball reproducible?~~ Answered: yes.** 134 artifacts held a byte-identical digest across two different pool revisions on a live upgrade ([§4](#4-the-versioned-pool)). Kept here only so a reader of an earlier draft can see it was closed by measurement. +- **How is a library's version accounted for in its consumers'?** `cozy-lib` is vendored into 36 artifacts, so one edit to it moved 25 otherwise-untouched packages. Does a consumer's version bump with its library — making 36 packages move whenever `cozy-lib` does, honestly but expensively — or does the library become a separately versioned package the consumer references, which is a delivery change? See [§4](#4-the-versioned-pool). - **Do `flux-plunger` and `flux-shard-operator` belong to core?** They exist only to make Flux deliver packages, which argues core; they are also Flux-version-coupled plumbing that a future delivery change would replace wholesale, which argues package. The classification in [§1](#1-four-tiers-one-package-model) puts them in core provisionally. - **Manifest kind and home.** A CRD applied to the cluster, a plain OCI artifact read by tooling, or both? #23's `TapIndex` cache is the obvious reader either way. - **How does the CalVer name relate to the existing `v1.x` stream?** Is `2026.08` a rename of the same stream, or does a final `v1.x` release announce the switch? What do existing release branches and backport automation do at the boundary? @@ -562,8 +596,13 @@ Collected from `main` on 2026-07-27 (post-`v1.6.0`) and re-measured on 2026-08-2 | `cozypkg` size / commits / documentation | 1811 lines / 3 commits / 0 lines of docs | `cmd/cozypkg/`, `git log` | | Core code churn, 12 months | ~620 commits across `internal/`, `pkg/`, `api/` | `git log --since='12 months ago'` | | Generated artifact revision derivation | content digest — the field is optional and cozystack never sets it | `fluxcd.yaml:522-528` (CRD doc), `packagesource_reconciler.go:241-244` (no `Revision`) | -| Packages moved by a patch release | 35 of 164 — 9 chart source, 13 image digest, **13 tag string only** — 129 untouched | `git diff v1.6.1..v1.6.2 -- packages/` | -| Packages moved by a minor release | 92 of 164 | `git diff v1.5.4..v1.6.0 -- packages/` | +| Packages moved by a patch release, from the diff | 35 of 164 — 9 chart source, 13 image digest, **13 tag string only** — 129 untouched | `git diff v1.6.1..v1.6.2 -- packages/` | +| Packages moved by a minor release, from the diff | 92 of 164 | `git diff v1.5.4..v1.6.0 -- packages/` | +| Artifacts moved by that patch release, **measured on a cluster** | 73 of 207 — 12 chart source, 18 image digest, **18 tag string only**, **25 `cozy-lib` fan-out** — 134 held | 3-node Talos stand, `isp-full`, 1.6.1 → 1.6.2, artifact digests before/after | +| HelmReleases and pods moved by it | 23 of 95 HelmReleases, **43 of 164 pods replaced** | same run | +| Pods restarted on a byte-identical image | Cilium 3, LINSTOR 8, MetalLB 4, objectstorage-controller 1 | same run; metallb pods came back on the digests both tags carry | +| Artifacts bundling a library chart | 36 of 207, all `cozy-lib`, copied in by the `ArtifactGenerator` | `kubectl get ag -A -o yaml` | +| source-watcher re-tar determinism | confirmed — 134 artifacts held a byte-identical digest across two pool revisions | same run | | Charts building from the root Go context | 11 (`COPY api pkg cmd internal`), so one `internal/` change moves every one of their digests | `packages/*/*/Makefile`, `images/*/Dockerfile` | | Flux archive determinism | byte-identical across builds; entries normalised to mtime `1970-01-01`, uid/gid `0` | `flux build artifact` twice on one tree, flux 2.8.6 | | `--reproducible` on the pool push | available, not passed | `packages/core/installer/Makefile:37` | From 76481563da1af4de6b7a5e140e0ff6047513d6b5 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Thu, 27 Aug 2026 18:36:43 +0500 Subject: [PATCH 07/13] docs(cozystack-as-a-distribution): test the fix, and find the library is now the bigger cause MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit "Drop the vendored tag and the churn goes away" was a claim, so it was run. Both release trees were reassembled the way the ArtifactGenerator does — package directory plus the library charts copied into it — and content-hashed twice: as shipped, and with the Cozystack version removed from every image reference. Unmodified, the model reproduces the cluster exactly at 73 moved and 134 held with the same split by cause, which is what licenses trusting it on the change it cannot observe. With the version gone from the refs: 59 moved, 148 held. The fourteen that stop are precisely the data plane — Cilium in all five variants, LINSTOR, MetalLB, Multus, kubeovn-plunger, Kamaji, linstor-gui, objectstorage-controller and seaweedfs-system — the same packages whose 16 pod restarts were measured, so the fix removes all of them. Four tag-only movers survive, and instructively: clickhouse, http-cache, mariadb and extra/seaweedfs stop moving for the tag and keep moving because they vendor cozy-lib. That reorders the work. Of the 59 survivors, 29 are library fan-out against 18 genuine rebuilds and 12 changed sources, so the library question stops being a footnote and becomes the largest single cause of churn left; the open question is relabelled accordingly. Fixing both takes the release from 73 moved artifacts to 30, each for a reason inside its own package. One implementation note the experiment produced: a first transform matching only `repo:tag@sha256:` recovered 4 of 18, because two common shapes put the tag elsewhere — a bare `tag: v1.6.1` with the digest in a sibling key, and `tag: v1.6.1@sha256:…` as a YAML value. A digest-only change has to cover every shape hack/lib/image-refs.sh enumerates, which is also the only place they should be enumerated. Assisted-By: Claude Signed-off-by: Myasnikov Daniil --- .../cozystack-as-a-distribution/README.md | 25 +++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-) diff --git a/design-proposals/cozystack-as-a-distribution/README.md b/design-proposals/cozystack-as-a-distribution/README.md index eaa3322..4088467 100644 --- a/design-proposals/cozystack-as-a-distribution/README.md +++ b/design-proposals/cozystack-as-a-distribution/README.md @@ -292,6 +292,25 @@ copy: This is not cosmetic and cannot be deleted the way the tag can — the library genuinely is part of the rendered chart. It is a statement about what a package *is*: a package's version must be a function of its own directory **and** the libraries it vendors, and a `cozy-lib` change legitimately bumps every consumer. Which makes `cozy-lib` a de-facto part of 36 packages' interface with no version on it — the same undeclared-interface problem [Testing](#testing) raises for cross-package `lookup`, appearing inside the first-party archive rather than between catalogs. Partial upgrades will not isolate a `cozy-lib` fix, and the design should say so rather than discover it later. +#### The counterfactual, also measured + +"Drop the tag and the churn goes away" is a claim, so it was tested rather than asserted. Both release trees were reassembled the way the `ArtifactGenerator` does — package directory plus the library charts copied into it — and content-hashed twice: as shipped, and with the Cozystack version removed from every image reference so the ref is `repo@sha256:…` alone. + +As shipped the model reproduces the cluster exactly: 73 moved, 134 held, with the same split by cause. That agreement is what licenses trusting it on the change it cannot observe directly. + +| | moved | held | +|---|---|---| +| as shipped | 73 | 134 | +| version removed from every vendored ref | **59** | **148** | + +**Fourteen artifacts stop moving outright, and they are precisely the data plane:** Cilium in all five variants, LINSTOR, MetalLB, Multus, kubeovn-plunger, Kamaji, linstor-gui, objectstorage-controller and seaweedfs-system. Those are the same packages whose 16 pod restarts were measured above, so the fix removes every one of them. + +Four tag-only movers survive — `apps/clickhouse`, `apps/http-cache`, `apps/mariadb`, `extra/seaweedfs` — and the reason is instructive rather than a shortfall: they stop moving *for the tag* and keep moving because they vendor `cozy-lib`, which changed in this release. + +Which reorders the remaining work. After the tag fix the 59 survivors are 29 library fan-out, 18 genuine image rebuilds and 12 changed chart sources — so **the library question is no longer a footnote, it is the largest single cause of churn left.** Fixing both would take the release from 73 moved artifacts to 30, every one of them for a reason inside the package. + +One implementation note the experiment produced. A first attempt at the transform matched only `repo:tag@sha256:…` and recovered 4 of the 18, because `hack/lib/image-refs.sh` documents five ref shapes and two of the common ones put the tag elsewhere — a bare `tag: v1.6.1` with the digest in a sibling key (Cilium), and `tag: v1.6.1@sha256:…` as a YAML value. A digest-only change has to cover every shape that file enumerates, and `image-refs.sh` is the right and only place to enumerate them. + #### Stop vendoring the tag **The tag fix is smaller than any other item in this proposal and independent of all of them.** Write `ghcr.io/cozystack/cozystack/metallb@sha256:…` and nothing else. The tag continues to exist in the registry — `hack/promote-retag.sh` pushes it — and the release manifest names it, so nothing that a human or a mirror needs is lost; only the copy that sits inside the chart and forces a reconcile goes away. `hack/promote-rewrite-tags.sh` and its bats suite are deleted outright, and the rc-leftover scan in `hack/verify-promoted-packages.sh` has nothing left to scan for. This is worth doing on its own merits whatever happens to the rest of the proposal, and its effect is measurable on the very next patch release. @@ -538,7 +557,7 @@ Both precedents carry a piece Cozystack does not have yet, and the model is only The ordering is chosen so that each phase is independently valuable and independently revertible, and so that the cheapest item with the largest measurable effect comes first. -1. **Stop vendoring the tag.** Pin first-party images by digest alone in every chart, delete `hack/promote-rewrite-tags.sh` and its bats suite, and pass `--reproducible` to the pool push. No design commitment of any kind. Measured on a cluster ([§4](#4-the-versioned-pool)), this removes 18 of the 73 artifact moves a patch release causes, and with them the Cilium, LINSTOR, MetalLB and objectstorage-controller pod restarts that deliver no new bytes. +1. **Stop vendoring the tag.** Pin first-party images by digest alone in every chart — covering all five ref shapes `hack/lib/image-refs.sh` enumerates, not just the inline one — delete `hack/promote-rewrite-tags.sh` and its bats suite, and pass `--reproducible` to the pool push. No design commitment of any kind. Measured against the real trees ([§4](#4-the-versioned-pool)), it takes a patch release from 73 moved artifacts to 59 and removes every one of the Cilium, LINSTOR, MetalLB and objectstorage-controller pod restarts, which deliver no new bytes. 2. **Cadence only.** Adopt the four-week train, publish the support window, cut `release-YYYY.MM` branches. No code changes. Two or three cycles of evidence before anything else depends on the cadence holding. 3. **Land the upgrade lane.** Get [cozystack#3276](https://github.com/cozystack/cozystack/pull/3276) merged and promote it from advisory to required for release PRs. Independently valuable — it is already finding upgrade-only defects nothing else reaches — and everything downstream of the manifest depends on being able to test an upgrade of one. 4. **Read the source-watcher archive path.** Confirm that `ArtifactGenerator`'s re-tar normalises entry metadata the way the `flux` CLI's does; the CLI half is already established in [§4](#4-the-versioned-pool). This is a code read, not a cluster experiment. @@ -554,7 +573,7 @@ The ordering is chosen so that each phase is independently valuable and independ ## Open questions - **~~Is the generated tarball reproducible?~~ Answered: yes.** 134 artifacts held a byte-identical digest across two different pool revisions on a live upgrade ([§4](#4-the-versioned-pool)). Kept here only so a reader of an earlier draft can see it was closed by measurement. -- **How is a library's version accounted for in its consumers'?** `cozy-lib` is vendored into 36 artifacts, so one edit to it moved 25 otherwise-untouched packages. Does a consumer's version bump with its library — making 36 packages move whenever `cozy-lib` does, honestly but expensively — or does the library become a separately versioned package the consumer references, which is a delivery change? See [§4](#4-the-versioned-pool). +- **How is a library's version accounted for in its consumers'? This is now the largest open item.** `cozy-lib` is vendored into 36 artifacts, so one edit to it moved 25 otherwise-untouched packages — and once the tag fix lands it accounts for 29 of the 59 remaining moves, more than any other cause ([§4](#4-the-versioned-pool)). Does a consumer's version bump with its library, which is honest but moves 36 packages whenever `cozy-lib` does? Or does the library become a separately versioned package the consumer references rather than embeds, which is a delivery change and the only route to isolating a `cozy-lib` fix in a partial upgrade? - **Do `flux-plunger` and `flux-shard-operator` belong to core?** They exist only to make Flux deliver packages, which argues core; they are also Flux-version-coupled plumbing that a future delivery change would replace wholesale, which argues package. The classification in [§1](#1-four-tiers-one-package-model) puts them in core provisionally. - **Manifest kind and home.** A CRD applied to the cluster, a plain OCI artifact read by tooling, or both? #23's `TapIndex` cache is the obvious reader either way. - **How does the CalVer name relate to the existing `v1.x` stream?** Is `2026.08` a rename of the same stream, or does a final `v1.x` release announce the switch? What do existing release branches and backport automation do at the boundary? @@ -603,6 +622,8 @@ Collected from `main` on 2026-07-27 (post-`v1.6.0`) and re-measured on 2026-08-2 | Pods restarted on a byte-identical image | Cilium 3, LINSTOR 8, MetalLB 4, objectstorage-controller 1 | same run; metallb pods came back on the digests both tags carry | | Artifacts bundling a library chart | 36 of 207, all `cozy-lib`, copied in by the `ArtifactGenerator` | `kubectl get ag -A -o yaml` | | source-watcher re-tar determinism | confirmed — 134 artifacts held a byte-identical digest across two pool revisions | same run | +| Counterfactual: version removed from every vendored ref | 59 moved / 148 held; the 14 eliminated are Cilium (5 variants), LINSTOR, MetalLB, Multus, kubeovn-plunger, Kamaji, linstor-gui, objectstorage-controller, seaweedfs-system | both trees reassembled as the ArtifactGenerator does and content-hashed; reproduces the cluster exactly when run unmodified | +| Largest cause remaining after that fix | `cozy-lib` fan-out, 29 of 59 | same model | | Charts building from the root Go context | 11 (`COPY api pkg cmd internal`), so one `internal/` change moves every one of their digests | `packages/*/*/Makefile`, `images/*/Dockerfile` | | Flux archive determinism | byte-identical across builds; entries normalised to mtime `1970-01-01`, uid/gid `0` | `flux build artifact` twice on one tree, flux 2.8.6 | | `--reproducible` on the pool push | available, not passed | `packages/core/installer/Makefile:37` | From 84e974f6f3c6541b4c10cc831420d8a07a372a6d Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Thu, 27 Aug 2026 21:36:09 +0500 Subject: [PATCH 08/13] docs(cozystack-as-a-distribution): confirm the tag fix end to end on a cluster MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The counterfactual had been modelled by content-hashing both trees. It has now been run: both release trees rebuilt with the Cozystack version removed from every image reference across all five ref shapes, pushed as two pool artifacts, and the same stand pointed at the first, settled, then pointed at the second. The model was exactly right. 59 artifacts moved against 73 as shipped, 148 held against 134, and the 14 that stopped are the 14 predicted with nothing newly moving. The residual splits as predicted too: 25 library fan-out, 18 image rebuilds, 12 changed sources, 4 tag-only survivors that move for cozy-lib rather than for the tag. The part worth having run: the data-plane restarts go to zero. Cilium 3 -> 0, LINSTOR 8 -> 0, MetalLB 4 -> 0, Multus 3 -> 0, objectstorage-controller 1 -> 0, and total pod churn halves from 43 of 164 to 21. HelmReleases taking a new revision fall from 23 to 16. One Kamaji pod still cycles, which its artifact no longer explains — a reschedule, not a delivery event. Rollout phase 1 and appendix A now carry the measured figures rather than the modelled ones. Assisted-By: Claude Signed-off-by: Myasnikov Daniil --- .../cozystack-as-a-distribution/README.md | 31 +++++++++++++------ 1 file changed, 21 insertions(+), 10 deletions(-) diff --git a/design-proposals/cozystack-as-a-distribution/README.md b/design-proposals/cozystack-as-a-distribution/README.md index 4088467..434fc62 100644 --- a/design-proposals/cozystack-as-a-distribution/README.md +++ b/design-proposals/cozystack-as-a-distribution/README.md @@ -292,24 +292,34 @@ copy: This is not cosmetic and cannot be deleted the way the tag can — the library genuinely is part of the rendered chart. It is a statement about what a package *is*: a package's version must be a function of its own directory **and** the libraries it vendors, and a `cozy-lib` change legitimately bumps every consumer. Which makes `cozy-lib` a de-facto part of 36 packages' interface with no version on it — the same undeclared-interface problem [Testing](#testing) raises for cross-package `lookup`, appearing inside the first-party archive rather than between catalogs. Partial upgrades will not isolate a `cozy-lib` fix, and the design should say so rather than discover it later. -#### The counterfactual, also measured +#### The counterfactual, run on the same cluster -"Drop the tag and the churn goes away" is a claim, so it was tested rather than asserted. Both release trees were reassembled the way the `ArtifactGenerator` does — package directory plus the library charts copied into it — and content-hashed twice: as shipped, and with the Cozystack version removed from every image reference so the ref is `repo@sha256:…` alone. +"Drop the tag and the churn goes away" is a claim, so it was run rather than asserted. Both release trees were rebuilt with the Cozystack version removed from every image reference — all five ref shapes, so the ref is `repo@sha256:…` alone — pushed as two pool artifacts, and the same stand was pointed at the first, allowed to settle, then pointed at the second. The measurement is that second transition, against the same 207 artifacts. -As shipped the model reproduces the cluster exactly: 73 moved, 134 held, with the same split by cause. That agreement is what licenses trusting it on the change it cannot observe directly. +| | artifacts moved | held | HelmReleases moved | pods replaced | +|---|---|---|---|---| +| as shipped, `v1.6.1` → `v1.6.2` | 73 | 134 | 23 / 95 | **43 / 164** | +| version removed from every vendored ref | **59** | **148** | 16 / 95 | **21 / 164** | -| | moved | held | +Exactly 14 artifacts stopped moving, exactly the 14 predicted, with nothing newly moving: Cilium in all six of its artifacts, LINSTOR, linstor-gui, MetalLB, Multus, kubeovn-plunger, Kamaji, objectstorage-controller and seaweedfs-system. The residual splits as predicted too — 25 library fan-out, 18 image rebuilds, 12 changed sources, 4 tag-only. + +**The data-plane restarts go to zero.** + +| pods replaced | as shipped | tag removed | |---|---|---| -| as shipped | 73 | 134 | -| version removed from every vendored ref | **59** | **148** | +| `cozy-cilium` | 3 | **0** | +| `cozy-linstor` | 8 | **0** | +| `cozy-metallb` | 4 | **0** | +| `cozy-multus` | 3 | **0** | +| `cozy-objectstorage-controller` | 1 | **0** | -**Fourteen artifacts stop moving outright, and they are precisely the data plane:** Cilium in all five variants, LINSTOR, MetalLB, Multus, kubeovn-plunger, Kamaji, linstor-gui, objectstorage-controller and seaweedfs-system. Those are the same packages whose 16 pod restarts were measured above, so the fix removes every one of them. +The CNI, the storage layer, the load balancer and the CNI multiplexer are not restarted at all once the version stops being vendored into the chart, and total pod churn halves. One Kamaji pod still cycles, which its artifact no longer explains — an ordinary reschedule rather than a delivery event. Four tag-only movers survive — `apps/clickhouse`, `apps/http-cache`, `apps/mariadb`, `extra/seaweedfs` — and the reason is instructive rather than a shortfall: they stop moving *for the tag* and keep moving because they vendor `cozy-lib`, which changed in this release. Which reorders the remaining work. After the tag fix the 59 survivors are 29 library fan-out, 18 genuine image rebuilds and 12 changed chart sources — so **the library question is no longer a footnote, it is the largest single cause of churn left.** Fixing both would take the release from 73 moved artifacts to 30, every one of them for a reason inside the package. -One implementation note the experiment produced. A first attempt at the transform matched only `repo:tag@sha256:…` and recovered 4 of the 18, because `hack/lib/image-refs.sh` documents five ref shapes and two of the common ones put the tag elsewhere — a bare `tag: v1.6.1` with the digest in a sibling key (Cilium), and `tag: v1.6.1@sha256:…` as a YAML value. A digest-only change has to cover every shape that file enumerates, and `image-refs.sh` is the right and only place to enumerate them. +One implementation note the experiment produced, and it is the trap this change will actually hit. A first attempt at the transform matched only `repo:tag@sha256:…` and recovered 4 of the 18, because `hack/lib/image-refs.sh` documents five ref shapes and two of the common ones put the tag elsewhere — a bare `tag: v1.6.1` with the digest in a sibling key (Cilium), and `tag: v1.6.1@sha256:…` as a YAML value. A digest-only change has to cover every shape that file enumerates, and `image-refs.sh` is the right and only place to enumerate them. #### Stop vendoring the tag @@ -557,7 +567,7 @@ Both precedents carry a piece Cozystack does not have yet, and the model is only The ordering is chosen so that each phase is independently valuable and independently revertible, and so that the cheapest item with the largest measurable effect comes first. -1. **Stop vendoring the tag.** Pin first-party images by digest alone in every chart — covering all five ref shapes `hack/lib/image-refs.sh` enumerates, not just the inline one — delete `hack/promote-rewrite-tags.sh` and its bats suite, and pass `--reproducible` to the pool push. No design commitment of any kind. Measured against the real trees ([§4](#4-the-versioned-pool)), it takes a patch release from 73 moved artifacts to 59 and removes every one of the Cilium, LINSTOR, MetalLB and objectstorage-controller pod restarts, which deliver no new bytes. +1. **Stop vendoring the tag.** Pin first-party images by digest alone in every chart — covering all five ref shapes `hack/lib/image-refs.sh` enumerates, not just the inline one — delete `hack/promote-rewrite-tags.sh` and its bats suite, and pass `--reproducible` to the pool push. No design commitment of any kind. Run end to end on a cluster ([§4](#4-the-versioned-pool)), it takes a patch release from 73 moved artifacts to 59 and from 43 replaced pods to 21, with the Cilium, LINSTOR, MetalLB, Multus and objectstorage-controller restarts going to zero. 2. **Cadence only.** Adopt the four-week train, publish the support window, cut `release-YYYY.MM` branches. No code changes. Two or three cycles of evidence before anything else depends on the cadence holding. 3. **Land the upgrade lane.** Get [cozystack#3276](https://github.com/cozystack/cozystack/pull/3276) merged and promote it from advisory to required for release PRs. Independently valuable — it is already finding upgrade-only defects nothing else reaches — and everything downstream of the manifest depends on being able to test an upgrade of one. 4. **Read the source-watcher archive path.** Confirm that `ArtifactGenerator`'s re-tar normalises entry metadata the way the `flux` CLI's does; the CLI half is already established in [§4](#4-the-versioned-pool). This is a code read, not a cluster experiment. @@ -622,7 +632,8 @@ Collected from `main` on 2026-07-27 (post-`v1.6.0`) and re-measured on 2026-08-2 | Pods restarted on a byte-identical image | Cilium 3, LINSTOR 8, MetalLB 4, objectstorage-controller 1 | same run; metallb pods came back on the digests both tags carry | | Artifacts bundling a library chart | 36 of 207, all `cozy-lib`, copied in by the `ArtifactGenerator` | `kubectl get ag -A -o yaml` | | source-watcher re-tar determinism | confirmed — 134 artifacts held a byte-identical digest across two pool revisions | same run | -| Counterfactual: version removed from every vendored ref | 59 moved / 148 held; the 14 eliminated are Cilium (5 variants), LINSTOR, MetalLB, Multus, kubeovn-plunger, Kamaji, linstor-gui, objectstorage-controller, seaweedfs-system | both trees reassembled as the ArtifactGenerator does and content-hashed; reproduces the cluster exactly when run unmodified | +| Counterfactual: version removed from every vendored ref | 59 moved / 148 held, 16/95 HelmReleases, **21/164 pods** (against 73 / 134 / 23 / 43); the 14 eliminated are Cilium (6 artifacts), LINSTOR, linstor-gui, MetalLB, Multus, kubeovn-plunger, Kamaji, objectstorage-controller, seaweedfs-system | same stand, two rebuilt pools pushed and applied in sequence | +| Data-plane pods restarted under that counterfactual | Cilium 0, LINSTOR 0, MetalLB 0, Multus 0, objectstorage-controller 0 | same run | | Largest cause remaining after that fix | `cozy-lib` fan-out, 29 of 59 | same model | | Charts building from the root Go context | 11 (`COPY api pkg cmd internal`), so one `internal/` change moves every one of their digests | `packages/*/*/Makefile`, `images/*/Dockerfile` | | Flux archive determinism | byte-identical across builds; entries normalised to mtime `1970-01-01`, uid/gid `0` | `flux build artifact` twice on one tree, flux 2.8.6 | From a73b2cb34276f239ba6dd191de1fa72bab3522c8 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Fri, 28 Aug 2026 10:01:16 +0500 Subject: [PATCH 09/13] docs(cozystack-as-a-distribution): resolve five open items against the tree MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Each of these was carrying a placeholder answer. Three turn out to be much smaller than the document claimed and two needed a different question. Extracting core needs no new machinery. The installer chart renders three files; everything else in the bootstrap lane is installed by the operator at startup from embedded manifests, twice already — crdinstall for the CRDs and fluxinstall for Flux. Moving the apiserver, controller and lineage webhook out of the pool is a third Install() beside those two, not a new delivery path, and ordering is already handled. The real work is handing ownership over from the platform Package without recreating the workloads, plus one coupling that was unstated: cozystack-controller rolls the apiserver DaemonSet on a config-hash, so if the apiserver moves and the controller does not, that loop crosses the tier boundary. cozy-lib has an answer, and it is not the one offered before. Helm requires a library chart to sit in the consumer's charts/, so referencing it across artifact boundaries is not available at all. The defect is that the copy always takes the head of the pool: Library is {Name, Path} with no version, the same shape defect as DependsOn, while apps/tenant already declares the dependency with version "*". Adding version to Library and resolving cozy-lib at it is the fix, and it makes the library the natural pilot for the versioned pool — one library, 36 consumers, visible edges, and 29 of the 59 moves that survive the tag fix. Added to the rollout as its own phase. Testing precondition 3 was overstated. 267 lookup sites across 35 charts is the raw count and most are the self-referential idiom; charts reading another package's CRDs number eleven, and the groups are enumerable. Eleven is a list somebody can read in an afternoon, so the ask is now concrete: declare those eleven and fail CI on a new one, which is #39's capability field doing the job it was proposed for. The precondition and #39 are the same work. The cadence section asked for two or three cycles of evidence before trusting a monthly train. The evidence is in the tags: the last four minors landed 27, 34 and 30 days apart, so monthly is already the observed cadence and writing it down ratifies rather than changes. The support window is likewise already chosen — backport.yaml enumerates release-X.Y branches and targets current plus previous — so the recommendation is to ratify two minor lines rather than invent a number that would need the bot changed first. The ApplicationDefinition consolidation is not about field names, which do not collide. It is about shape: #6's versions[] decides whether every other field lives at the top level or inside a version, so it is one decision, it is #6's, and the rest sequence behind it. Assisted-By: Claude Signed-off-by: Myasnikov Daniil --- .../cozystack-as-a-distribution/README.md | 53 +++++++++++++------ 1 file changed, 38 insertions(+), 15 deletions(-) diff --git a/design-proposals/cozystack-as-a-distribution/README.md b/design-proposals/cozystack-as-a-distribution/README.md index 434fc62..70834bf 100644 --- a/design-proposals/cozystack-as-a-distribution/README.md +++ b/design-proposals/cozystack-as-a-distribution/README.md @@ -160,7 +160,13 @@ Everything else under `packages/{system,apps,extra}` is a package without argume #### Two consequences the rest of this proposal depends on -**The delivery boundary is in the wrong place today, and moving it is real work.** Only two things currently ship outside the package pool: the CRDs, concatenated from `internal/crdinstall/manifests/*.yaml` into `_out/assets/cozystack-crds.yaml`, and the operator, rendered from `packages/core/installer` by the root `manifests` target. Flux itself comes from `internal/fluxinstall/manifests/fluxcd.yaml`. Everything else in the table above — the apiserver, the controller, the lineage webhook — travels through the pool as ordinary `packages/system/*` charts, on the same path as Cilium. Declaring them core therefore means moving three charts from the pool into the bootstrap lane, which changes how they are installed, upgraded and rolled back. This is the largest single piece of unscoped work in the proposal and it should be planned as its own change, not folded into the versioning work. +**The delivery boundary is in the wrong place today, but moving it needs no new machinery.** The installer chart renders exactly three things — the operator Deployment, the `cozy-system` namespace and its labels. Everything else in the bootstrap lane is installed by the operator itself at startup, from embedded manifests: `crdinstall.Install` for the `cozystack.io` CRDs, `fluxinstall.Install` for Flux, then `installPlatformPackageSource`. Everything after that travels through the pool, including the apiserver, the controller and the lineage webhook, which are ordinary `packages/system/*` charts on the same path as Cilium. + +So declaring them core means a third `Install()` from embedded manifests alongside the two that already work, not a new delivery path. Ordering is already handled — the CRDs go in before the `PackageSource` exists. + +The real work is the ownership migration rather than the mechanism. Those three are HelmReleases owned by the platform `Package` today, so the change has to hand ownership to the operator without recreating the workload; recreating the apiserver mid-upgrade is the failure this must not have. `helm.sh/resource-policy: keep` is already on the bundle-emitted `Package` CRs, which is the right half of the primitive, but the handover itself has to be designed. + +One coupling to resolve in the same change: `cozystack-controller` rolls the apiserver DaemonSet by updating `cozystack.io/config-hash` when `ApplicationDefinition` objects change. If the apiserver moves to the bootstrap lane and the controller that rolls it stays a package, that control loop crosses the tier boundary. Either both move, or the rollout responsibility moves with the apiserver. A naming collision comes with it: `packages/core/` today holds `flux-aio`, `installer`, `platform`, `talos` and `testing`, of which only `platform` and part of `installer` are core in the sense used here. The directory name will have to give way, most likely by the same argument #39 makes — the tier is a declared field, not a path. @@ -290,7 +296,15 @@ copy: 36 of the 207 artifacts bundle `cozy-lib` this way, and `packages/library/cozy-lib/templates/_barman.tpl` changed in this release. One edit to one library template therefore moved 25 packages, every one of which is otherwise untouched. -This is not cosmetic and cannot be deleted the way the tag can — the library genuinely is part of the rendered chart. It is a statement about what a package *is*: a package's version must be a function of its own directory **and** the libraries it vendors, and a `cozy-lib` change legitimately bumps every consumer. Which makes `cozy-lib` a de-facto part of 36 packages' interface with no version on it — the same undeclared-interface problem [Testing](#testing) raises for cross-package `lookup`, appearing inside the first-party archive rather than between catalogs. Partial upgrades will not isolate a `cozy-lib` fix, and the design should say so rather than discover it later. +This is not cosmetic and cannot be deleted the way the tag can — the library genuinely is part of the rendered chart, and Helm offers no way out: a library chart has to be present in the consumer's `charts/`, so "reference it instead of embedding it" is not an available option across artifact boundaries. + +The defect is not the copy, it is that the copy always takes the head of the pool. `Library` in the data model is `{Name, Path}` (`api/v1alpha1/packagesource_types.go:86-94`) — a path with no version, the same shape defect as `DependsOn`. The consumer side already has the declaration and has nothing to put in it: `packages/apps/tenant/Chart.yaml` carries `dependencies: [{name: cozy-lib, version: "*", repository: "file://charts/cozy-lib"}]`. + +**Adding a version to `Library` is the fix**, with the operator resolving `library/cozy-lib` at that version rather than at the pool head. A consumer then moves when it takes a new library, not when the library changes. + +That also makes `cozy-lib` the natural pilot for the versioned pool in Option A above, and a far better one than versioning all 164 packages at once: one library, 36 consumers, edges that are already visible in the `ArtifactGenerator`'s copy operations, and a measurable result — 29 of the 59 moves that survive the tag fix. It is the smallest change that exercises the whole mechanism end to end. + +Until then `cozy-lib` is a de-facto part of 36 packages' interface with no version on it, which is the same undeclared-interface problem [Testing](#testing) raises for cross-package `lookup`, inside the first-party archive rather than between catalogs. #### The counterfactual, run on the same cluster @@ -444,6 +458,8 @@ Two practical notes for whoever reconciles these. First, adopting per-package ve Three proposals are independently adding fields to the same object, and it is worth naming what it is becoming. `ApplicationDefinition` already carries kind, plural/singular, the OpenAPI schema, the release prefix and `chartRef`, dashboard metadata, and secret/ingress projections. #39 adds visibility, cardinality, protection, and capability provides/consumes. #6 adds `versions[]` with a storage version and conversion templates. cozystack#3448 adds a group selector. This proposal adds the package version, its section, its dependency constraints, and `requiresCore`. +The four sets do not collide on names, so the coordination this needs is narrower than it looks and also more urgent. What they collide on is **shape**: #6 introduces `versions[]`, which decides whether every other field lives at the top level or inside a version. If #6 lands last, everything added before it moves. So the consolidating pass is really one decision — does `ApplicationDefinition` become versioned before anyone else adds a top-level field — and it is #6's to make, with the rest sequenced behind it. + That is a Debian `control` file. Recognising it has one practical consequence worth acting on now: the object is accumulating fields from four directions at `v1alpha1`, external catalogs are about to depend on its shape across repository boundaries (#43 flags exactly this), and nobody owns its coherence. **This proposal recommends a single consolidating pass on `ApplicationDefinition` — one PR, one shape, all four proposals' fields reviewed together — before any of them ships its fields independently.** Otherwise the cross-repo contract that #43 needs stabilised gets four uncoordinated `v1alpha1` extensions first. ### 10. Bundles stay, and gain versions @@ -454,7 +470,9 @@ Worth noting for reviewers that the bundle layer already carries the correct sem ## Release cadence and branch model -The cadence change is separable from everything above and should ship first, because it delivers most of the operational benefit at almost no implementation cost and produces two or three cycles of evidence before the versioning work bets on it. +The cadence change is separable from everything above and costs almost nothing to adopt, for a reason worth stating plainly: **monthly is already the observed cadence.** The last four minor releases landed on 2026-04-22, 2026-05-19, 2026-06-22 and 2026-07-22 — 27, 34 and 30 days apart. Writing the train down ratifies what the project already does rather than asking it to change, so the risk here is close to zero and the "two or three cycles of evidence" an earlier draft asked for are already in the tags. + +What is genuinely new is only the shape inside the month, and the rule that a train which is not ready skips rather than slips. A four-week train: @@ -481,7 +499,7 @@ Three rules make it hold: 1. **A train that is not ready skips the month; it never extends.** A calendar name makes slipping impossible to hide, and v1.6.0 needed four release candidates with a NO-GO on rc.1 for upgrade-only blockers. `2026.09` not existing is a clean, legible outcome. `2026.08` shipping on 12 September is not. 2. **Packages do not get release branches.** Only the train does. Packages are released from `main` and referenced by version in the manifest. Otherwise the branch topology multiplies by the number of packages. -3. **The support window is published before the cadence changes.** Twelve releases a year cannot each be supported. The proposal recommends stating it explicitly — for example, the current release plus the two preceding, with security fixes only for the older two — and making `supportedUpgradeFrom` in the manifest the machine-readable form of that promise. +3. **The support window is published before the cadence changes,** and the tooling has already chosen it. `.github/workflows/backport.yaml` enumerates the real `release-X.Y` branches, sorts them descending and targets the current and the previous one, so **the de-facto window today is two minor lines** — implemented, not merely intended. The proposal recommends ratifying that rather than inventing a different number: current minor plus the one before it, security-only for the older. Any other answer means changing the backport automation first, which is a cost that should be paid deliberately if at all. `supportedUpgradeFrom` in the manifest then becomes the machine-readable form of a promise the repository is already keeping. ### Core releases between trains @@ -547,7 +565,11 @@ Both precedents carry a piece Cozystack does not have yet, and the model is only 3. **Package-to-package edges must become visible, or the guarantee has holes it cannot see.** This is the gap that has no precedent-supplied answer, and it is [an open question below](#open-questions) promoted to a prerequisite. Debian can compute which reverse-dependencies to re-test because `Depends: libfoo (>= 1.2)` is a versioned edge. Cozystack's equivalent is `DependsOn []string` (`api/v1alpha1/packagesource_types.go:74`, and the per-component form at `:135`) — 52 declared edges carrying names and no constraints, which orders installation and says nothing about compatibility. - The larger part is not declared at all. Charts read each other's live state: of 267 `lookup` call sites across 35 charts, most are the ordinary self-referential idiom, but several reach into another package's CRDs — `postgresql.cnpg.io/v1` `Cluster` (5), `instancetype.kubevirt.io/v1beta1` `VirtualMachineClusterInstancetype` (5), `cozystack.io/v1alpha1` `Package` (2), plus `MachineSet`, `DataVolume`, `BucketClaim` and `StorageClass`. A chart's *rendering* therefore depends on a CRD version another package installs. `requiresCore` cannot express that — the dependency is not on core. A values-schema diff cannot detect a break in it — the schema did not change. The `lookup` simply returns empty and the template renders something else, silently. + The larger part is not declared at all, though it is smaller than the raw count suggests. There are 267 `lookup` call sites across 35 charts, and the great majority are the ordinary self-referential idiom — a chart reading its own `Secret` to preserve a generated password across an upgrade. **Charts that read another package's CRDs number eleven**: `apps/kubernetes`, `apps/kubernetes-nodes`, `apps/tenant`, `apps/vm-disk`, `apps/vm-instance`, `core/platform`, `system/backupstrategy-controller`, `system/harbor`, `system/keycloak`, `system/monitoring`, `system/seaweedfs-db`. The groups they reach into are equally enumerable — `postgresql.cnpg.io`, `kubevirt.io` with `instancetype` and `cdi`, `cluster.x-k8s.io`, `cozystack.io`, `storage.k8s.io`. + + For those eleven a chart's *rendering* depends on a CRD version another package installs. `requiresCore` cannot express it, because the dependency is not on core. A values-schema diff cannot detect a break in it, because the schema did not change. The `lookup` returns empty and the template renders something else, silently. + + Eleven is a list somebody can read in an afternoon, which makes the ask concrete rather than open-ended: **declare the existing eleven and fail CI on a new one.** A cross-package `lookup` outside core becomes an error unless the package declares what it consumes, which is #39's capability field doing exactly the job it was proposed for — so this precondition and #39 are the same work, not two. The first-party archive already has an instance of this, measured rather than hypothesised: `cozy-lib` is copied into 36 artifacts, and one edit to it moved 25 packages on a real upgrade ([§4](#4-the-versioned-pool)). That edge is at least *visible* in the `ArtifactGenerator`'s copy operations, which the `lookup` edges are not — so it is the tractable half of the same problem and the natural place to start. @@ -571,23 +593,24 @@ The ordering is chosen so that each phase is independently valuable and independ 2. **Cadence only.** Adopt the four-week train, publish the support window, cut `release-YYYY.MM` branches. No code changes. Two or three cycles of evidence before anything else depends on the cadence holding. 3. **Land the upgrade lane.** Get [cozystack#3276](https://github.com/cozystack/cozystack/pull/3276) merged and promote it from advisory to required for release PRs. Independently valuable — it is already finding upgrade-only defects nothing else reaches — and everything downstream of the manifest depends on being able to test an upgrade of one. 4. **Read the source-watcher archive path.** Confirm that `ArtifactGenerator`'s re-tar normalises entry metadata the way the `flux` CLI's does; the CLI half is already established in [§4](#4-the-versioned-pool). This is a code read, not a cluster experiment. -5. **Extract core.** Move `cozystack-api`, `cozystack-controller` and `lineage-controller-webhook` out of the package pool into the bootstrap lane alongside the operator and the CRDs, and resolve the `packages/core/` naming collision (see [§1](#1-four-tiers-one-package-model)). This is the largest piece of work in the list and it is deliberately placed before any versioning change, because a tier boundary that delivery does not honour cannot carry a version. -6. **Manifest, inert.** Publish the release manifest with all versions equal to the current platform version. Add `version` to the data model. Nothing resolves differently. Add the two CI gates. -7. **ApplicationDefinition consolidation.** One coordinated pass folding this proposal's fields with #39's, #6's, and #3448's, before external catalogs depend on the shape. -8. **Migration framework.** `cozy-lib` migration helper; port the four existing ad-hoc hooks onto it; new migrations are authored per-package; the global lane is frozen except for cross-cutting cases. -9. **First real partial upgrade.** One release where the manifest moves a small number of low-risk packages (`bucket`, `http-cache` — not `postgres`, not `kubernetes`) and everything else holds. Measure what the tooling misses. -10. **`cozypkg`.** Versions, search, show, upgrade, hold, non-interactive output, and documentation. -11. **Repositories.** #43's catalogs and #18's taps land on the now-versioned mechanism; the manifest's `repositories` list ships the org catalog enabled-by-default per #43's shim plan. -12. **CalVer switch.** Rename the release stream once the manifest, partial upgrades, and the support window are all proven. Last, not first. +5. **Extract core.** Move `cozystack-api`, `cozystack-controller` and `lineage-controller-webhook` out of the package pool into the bootstrap lane alongside the operator and the CRDs, as a third embedded-manifest `Install()` next to `crdinstall` and `fluxinstall`, and resolve the `packages/core/` naming collision (see [§1](#1-four-tiers-one-package-model)). The mechanism is not new; the work is handing ownership over from the platform `Package` without recreating the workloads, plus deciding where the config-hash rollout lives. Deliberately placed before any versioning change, because a tier boundary that delivery does not honour cannot carry a version. +6. **Version the library.** Add `version` to `Library`, resolve `cozy-lib` at it, and let consumers move when they take a new one. One library and 36 consumers is the smallest thing that exercises the versioned pool end to end, and it is worth 29 of the 59 artifact moves that survive phase 1 ([§4](#4-the-versioned-pool)). +7. **Manifest, inert.** Publish the release manifest with all versions equal to the current platform version. Add `version` to the data model. Nothing resolves differently. Add the two CI gates. +8. **ApplicationDefinition consolidation.** One coordinated pass folding this proposal's fields with #39's, #6's, and #3448's, before external catalogs depend on the shape. +9. **Migration framework.** `cozy-lib` migration helper; port the four existing ad-hoc hooks onto it; new migrations are authored per-package; the global lane is frozen except for cross-cutting cases. +10. **First real partial upgrade.** One release where the manifest moves a small number of low-risk packages (`bucket`, `http-cache` — not `postgres`, not `kubernetes`) and everything else holds. Measure what the tooling misses. +11. **`cozypkg`.** Versions, search, show, upgrade, hold, non-interactive output, and documentation. +12. **Repositories.** #43's catalogs and #18's taps land on the now-versioned mechanism; the manifest's `repositories` list ships the org catalog enabled-by-default per #43's shim plan. +13. **CalVer switch.** Rename the release stream once the manifest, partial upgrades, and the support window are all proven. Last, not first. ## Open questions - **~~Is the generated tarball reproducible?~~ Answered: yes.** 134 artifacts held a byte-identical digest across two different pool revisions on a live upgrade ([§4](#4-the-versioned-pool)). Kept here only so a reader of an earlier draft can see it was closed by measurement. -- **How is a library's version accounted for in its consumers'? This is now the largest open item.** `cozy-lib` is vendored into 36 artifacts, so one edit to it moved 25 otherwise-untouched packages — and once the tag fix lands it accounts for 29 of the 59 remaining moves, more than any other cause ([§4](#4-the-versioned-pool)). Does a consumer's version bump with its library, which is honest but moves 36 packages whenever `cozy-lib` does? Or does the library become a separately versioned package the consumer references rather than embeds, which is a delivery change and the only route to isolating a `cozy-lib` fix in a partial upgrade? +- **When does a consumer take a new library?** Adding `version` to `Library` is the mechanism ([§4](#4-the-versioned-pool)); the policy is open. Do consumers pin and bump individually, which is what isolates a `cozy-lib` fix but means 36 deliberate bumps to roll one out everywhere, or does the platform bump them together at each train, which is simpler and gives back most of the isolation? Renovate-style automation makes the first tractable, so this is a question about who owns the bump rather than about whether it is affordable. - **Do `flux-plunger` and `flux-shard-operator` belong to core?** They exist only to make Flux deliver packages, which argues core; they are also Flux-version-coupled plumbing that a future delivery change would replace wholesale, which argues package. The classification in [§1](#1-four-tiers-one-package-model) puts them in core provisionally. - **Manifest kind and home.** A CRD applied to the cluster, a plain OCI artifact read by tooling, or both? #23's `TapIndex` cache is the obvious reader either way. - **How does the CalVer name relate to the existing `v1.x` stream?** Is `2026.08` a rename of the same stream, or does a final `v1.x` release announce the switch? What do existing release branches and backport automation do at the boundary? -- **Support window length.** Three releases? Six? Security-only tail? This must be answered before the cadence changes in rollout phase 2, not after. +- **Support window length.** The backport automation already implements current-plus-previous, so the open part is whether to ratify that or widen it, and widening means changing the bot first. This must be settled before the cadence is written down in rollout phase 2, not after. - **Does core ship between trains?** Recommendation above is no; maintainers should confirm. - **Version reporting and diagnostics.** With holds and partial upgrades, a cluster's state is a version vector rather than a single string. `cozypkg`, the dashboard, and the diagnostic bundle (`cozyreport` / crust-gather) must all carry it, or support gets harder rather than easier. Who owns that surface? - **Inter-package compatibility — promoted to a prerequisite, still unanswered.** Debian works because packages have declared ABIs. Helm charts have none: `DependsOn` is a list of names with no version, and the real interface is the values schema plus what one chart `lookup`s about another's live state — which 35 charts do across 267 call sites, several of them reaching into another package's CRDs. Should cross-package `lookup` be forbidden outside core, or should packages declare the interfaces they consume with a version? [Testing](#testing) explains why the manifest's guarantee is incomplete until this is settled; it does not settle it. From 1882040876885369b8f60746ab3f817cf3b0c76a Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Fri, 28 Aug 2026 11:01:05 +0500 Subject: [PATCH 10/13] docs(cozystack-as-a-distribution): correct how CRDs reach the cluster The previous commit said everything in the bootstrap lane past the operator Deployment is installed by the operator at startup. True on the Helm path, silent about the other one. The installer chart carries no crds/ directory and no CRD hook; its only pre-install hook is the Job that labels cozy-system. It passes --install-crds=true and --install-flux=true to the operator, both of which default to false in the binary, so on the Helm path the operator does install them from embedded manifests. On the kubectl apply path the same manifests ship as a release asset: make manifests concatenates internal/crdinstall/manifests/*.yaml into _out/assets/cozystack-crds.yaml and upload-assets.sh puts it in the release beside the rendered operator. One source, two deliveries. That adds a requirement to the core extraction which the two existing installs answer implicitly: anything moved into the bootstrap lane has to be present on both paths, or the Helm install and the kubectl apply install stop containing the same platform. The apiserver would need a release-asset entry, not only an Install(). Assisted-By: Claude Signed-off-by: Myasnikov Daniil --- design-proposals/cozystack-as-a-distribution/README.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/design-proposals/cozystack-as-a-distribution/README.md b/design-proposals/cozystack-as-a-distribution/README.md index 70834bf..bf6335b 100644 --- a/design-proposals/cozystack-as-a-distribution/README.md +++ b/design-proposals/cozystack-as-a-distribution/README.md @@ -160,9 +160,13 @@ Everything else under `packages/{system,apps,extra}` is a package without argume #### Two consequences the rest of this proposal depends on -**The delivery boundary is in the wrong place today, but moving it needs no new machinery.** The installer chart renders exactly three things — the operator Deployment, the `cozy-system` namespace and its labels. Everything else in the bootstrap lane is installed by the operator itself at startup, from embedded manifests: `crdinstall.Install` for the `cozystack.io` CRDs, `fluxinstall.Install` for Flux, then `installPlatformPackageSource`. Everything after that travels through the pool, including the apiserver, the controller and the lineage webhook, which are ordinary `packages/system/*` charts on the same path as Cilium. +**The delivery boundary is in the wrong place today, but moving it needs no new machinery.** The installer chart renders exactly three things — the operator Deployment, the `cozy-system` namespace, and a pre-install Job that labels it. It carries no `crds/` directory and no CRD hook. What installs the CRDs depends on the path: the chart passes `--install-crds=true` and `--install-flux=true` to the operator (both default to `false` in the binary), so on the Helm path the operator installs them itself at startup from embedded manifests, via `crdinstall.Install` and `fluxinstall.Install`, then creates the platform `PackageSource`. On the `kubectl apply` path the same manifests ship as a release asset — `make manifests` concatenates `internal/crdinstall/manifests/*.yaml` into `_out/assets/cozystack-crds.yaml` and `hack/upload-assets.sh` uploads it beside the rendered operator — and the operator applies them idempotently afterwards. One source, two deliveries. -So declaring them core means a third `Install()` from embedded manifests alongside the two that already work, not a new delivery path. Ordering is already handled — the CRDs go in before the `PackageSource` exists. +Everything after that travels through the pool, including the apiserver, the controller and the lineage webhook, which are ordinary `packages/system/*` charts on the same path as Cilium. + +So declaring them core means a third embedded-manifest `Install()` alongside the two that already work, not a new delivery path. Ordering is already handled: the CRDs go in before the `PackageSource` exists. + +It does add one decision the two existing installs already answer implicitly. Anything moved into the bootstrap lane has to be present on *both* paths, or the Helm install and the `kubectl apply` install stop containing the same platform. The apiserver would need its own entry in the release assets, not only its `Install()`. The real work is the ownership migration rather than the mechanism. Those three are HelmReleases owned by the platform `Package` today, so the change has to hand ownership to the operator without recreating the workload; recreating the apiserver mid-upgrade is the failure this must not have. `helm.sh/resource-policy: keep` is already on the bundle-emitted `Package` CRs, which is the right half of the primitive, but the handover itself has to be designed. From db47b07442eb394d66907e7d9ee6f12e40d3cd3d Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Fri, 28 Aug 2026 11:36:23 +0500 Subject: [PATCH 11/13] docs(cozystack-as-a-distribution): audit cozy-lib, and add shrinking it as its own phase Versioning Library is the structural fix for the fan-out. Counting what is actually in the library turns up a cheaper one that, for this release, would have been enough on its own. cozy-lib is 728 lines and 43 helpers, and the consumer counts are very uneven. Two helpers carry 48 of roughly 55 relationships: the tenant RBAC subject model at 27 consumers and the resource-preset model at 21, both genuinely cross-cutting and both belonging there. Against that, _tls.tpl has no caller at all and says so in its own comment, 14 of the 15 helpers in _cozyconfig.tpl are exercised only by the library's test chart (their apparent in-library callers are Usage: lines in doc comments), _strings.tpl is three lines for one consumer, and rbac.subjectsForTenant has no caller of either kind. _barman.tpl explains the whole event. Its two consumers are apps/postgres and system/keycloak, and keycloak is there because it runs a CNPG Postgres of its own, so the library is being used as a private channel between two packages rather than as a shared abstraction. That is the package-to-package edge Testing worries about, wearing a library's clothes and so invisible to anything looking for one. And it was the only change to cozy-lib in this release: 25 packages moved because a 24-line template with two consumers landed in a chart 36 packages embed. So the two fixes compose. Versioning decides when a consumer takes a change, shrinking decides how often there is one. The four candidates for eviction are 265 of the 728 lines and they change for reasons that concern one or two packages. Added to the rollout ahead of versioning the library, on the same grounds as the tag fix: hygiene, measurable, no design commitment. Assisted-By: Claude Signed-off-by: Myasnikov Daniil --- .../cozystack-as-a-distribution/README.md | 45 +++++++++++++++---- 1 file changed, 37 insertions(+), 8 deletions(-) diff --git a/design-proposals/cozystack-as-a-distribution/README.md b/design-proposals/cozystack-as-a-distribution/README.md index bf6335b..3d8d38a 100644 --- a/design-proposals/cozystack-as-a-distribution/README.md +++ b/design-proposals/cozystack-as-a-distribution/README.md @@ -310,6 +310,31 @@ That also makes `cozy-lib` the natural pilot for the versioned pool in Option A Until then `cozy-lib` is a de-facto part of 36 packages' interface with no version on it, which is the same undeclared-interface problem [Testing](#testing) raises for cross-package `lookup`, inside the first-party archive rather than between catalogs. +#### The library is also carrying things that are not shared + +Versioning `Library` is the structural fix. There is a cheaper one next to it, and for this particular release it would have been sufficient on its own. + +`cozy-lib` is 728 lines and 43 helpers, and its consumer counts are extremely uneven. Excluding the library itself and its own test chart: + +| File | Lines | Consumers outside the library | +|---|---|---| +| `_resources.tpl` | 224 | `defaultingSanitize` 21, `toFloat` 5, `sanitize` 4, `flatten` 2, `javaHeap` 1 | +| `_rbac.tpl` | 106 | `subjectsForTenantAndAccessLevel` 27 | +| `_resourcepresets.tpl` | 98 | none directly; the preset table `_resources.tpl` reads | +| `_network.tpl` | 23 | `disableLoadBalancerNodePorts` 6 | +| `_cozyconfig.tpl` | 153 | `image` 3. The other 14 helpers have **no caller at all** | +| `_barman.tpl` | 24 | 2 — `apps/postgres` and `system/keycloak` | +| `_strings.tpl` | 3 | 1 — `apps/vpc` | +| `_tls.tpl` | 92 | **none** | + +Two helpers out of 43 carry 48 of the roughly 55 consumer relationships, and they are the two that deserve to be there: the tenant RBAC subject model and the resource-preset model, both genuinely cross-cutting. `_tls.tpl` says of itself "there is no production caller yet - this is the pattern one uses". The public surface of `_cozyconfig.tpl` — `root-host`, `bundle-name`, the `ns-*` accessors, the `ipv4-*` CIDRs, `branding`, `scheduling` — is exercised only by `tests/cozy-lib-tests`; its apparent in-library callers are `Usage:` lines inside doc comments. `cozy-lib.rbac.subjectsForTenant` has no caller of either kind. + +`_barman.tpl` is the interesting one, because it explains the whole event. Its two consumers are `apps/postgres` and `system/keycloak`, and keycloak is there because it runs a CNPG Postgres of its own. So the library is acting as a private channel between two packages rather than as a shared abstraction — the same package-to-package edge [Testing](#testing) worries about, wearing a library's clothes and therefore invisible to anything looking for one. + +And it is the whole of the fan-out. The only change to `cozy-lib` in this release was `A packages/library/cozy-lib/templates/_barman.tpl`. **Twenty-five packages moved because a 24-line template with two consumers was added to a chart 36 packages embed.** Had it lived in `apps/postgres`, the library would have moved nothing. + +So the two fixes compose rather than compete. Versioning `Library` decides *when* a consumer takes a change; shrinking the library decides *how often there is one to take*. `_tls.tpl`, `_strings.tpl`, `_barman.tpl` and `_cozyconfig.tpl` minus `image` are 265 of the 728 lines, and they change for reasons that concern one or two packages rather than 36. Moving them back to their consumers is hygiene with no design commitment attached, and it is worth doing first for the same reason the tag fix is. + #### The counterfactual, run on the same cluster "Drop the tag and the churn goes away" is a claim, so it was run rather than asserted. Both release trees were rebuilt with the Cozystack version removed from every image reference — all five ref shapes, so the ref is `repo@sha256:…` alone — pushed as two pool artifacts, and the same stand was pointed at the first, allowed to settle, then pointed at the second. The measurement is that second transition, against the same 207 artifacts. @@ -598,14 +623,15 @@ The ordering is chosen so that each phase is independently valuable and independ 3. **Land the upgrade lane.** Get [cozystack#3276](https://github.com/cozystack/cozystack/pull/3276) merged and promote it from advisory to required for release PRs. Independently valuable — it is already finding upgrade-only defects nothing else reaches — and everything downstream of the manifest depends on being able to test an upgrade of one. 4. **Read the source-watcher archive path.** Confirm that `ArtifactGenerator`'s re-tar normalises entry metadata the way the `flux` CLI's does; the CLI half is already established in [§4](#4-the-versioned-pool). This is a code read, not a cluster experiment. 5. **Extract core.** Move `cozystack-api`, `cozystack-controller` and `lineage-controller-webhook` out of the package pool into the bootstrap lane alongside the operator and the CRDs, as a third embedded-manifest `Install()` next to `crdinstall` and `fluxinstall`, and resolve the `packages/core/` naming collision (see [§1](#1-four-tiers-one-package-model)). The mechanism is not new; the work is handing ownership over from the platform `Package` without recreating the workloads, plus deciding where the config-hash rollout lives. Deliberately placed before any versioning change, because a tier boundary that delivery does not honour cannot carry a version. -6. **Version the library.** Add `version` to `Library`, resolve `cozy-lib` at it, and let consumers move when they take a new one. One library and 36 consumers is the smallest thing that exercises the versioned pool end to end, and it is worth 29 of the 59 artifact moves that survive phase 1 ([§4](#4-the-versioned-pool)). -7. **Manifest, inert.** Publish the release manifest with all versions equal to the current platform version. Add `version` to the data model. Nothing resolves differently. Add the two CI gates. -8. **ApplicationDefinition consolidation.** One coordinated pass folding this proposal's fields with #39's, #6's, and #3448's, before external catalogs depend on the shape. -9. **Migration framework.** `cozy-lib` migration helper; port the four existing ad-hoc hooks onto it; new migrations are authored per-package; the global lane is frozen except for cross-cutting cases. -10. **First real partial upgrade.** One release where the manifest moves a small number of low-risk packages (`bucket`, `http-cache` — not `postgres`, not `kubernetes`) and everything else holds. Measure what the tooling misses. -11. **`cozypkg`.** Versions, search, show, upgrade, hold, non-interactive output, and documentation. -12. **Repositories.** #43's catalogs and #18's taps land on the now-versioned mechanism; the manifest's `repositories` list ships the org catalog enabled-by-default per #43's shim plan. -13. **CalVer switch.** Rename the release stream once the manifest, partial upgrades, and the support window are all proven. Last, not first. +6. **Shrink the library.** Move `_barman.tpl` to its two consumers, `_strings.tpl` to `apps/vpc`, delete `_tls.tpl` and the uncalled surface of `_cozyconfig.tpl`, and drop `cozy-lib.rbac.subjectsForTenant`. Hygiene, no design commitment, and it removes the class of change that caused this release's 25-package fan-out ([§4](#4-the-versioned-pool)). +7. **Version the library.** Add `version` to `Library`, resolve `cozy-lib` at it, and let consumers move when they take a new one. One library and 36 consumers is the smallest thing that exercises the versioned pool end to end, and it is worth 29 of the 59 artifact moves that survive phase 1. +8. **Manifest, inert.** Publish the release manifest with all versions equal to the current platform version. Add `version` to the data model. Nothing resolves differently. Add the two CI gates. +9. **ApplicationDefinition consolidation.** One coordinated pass folding this proposal's fields with #39's, #6's, and #3448's, before external catalogs depend on the shape. +10. **Migration framework.** `cozy-lib` migration helper; port the four existing ad-hoc hooks onto it; new migrations are authored per-package; the global lane is frozen except for cross-cutting cases. +11. **First real partial upgrade.** One release where the manifest moves a small number of low-risk packages (`bucket`, `http-cache` — not `postgres`, not `kubernetes`) and everything else holds. Measure what the tooling misses. +12. **`cozypkg`.** Versions, search, show, upgrade, hold, non-interactive output, and documentation. +13. **Repositories.** #43's catalogs and #18's taps land on the now-versioned mechanism; the manifest's `repositories` list ships the org catalog enabled-by-default per #43's shim plan. +14. **CalVer switch.** Rename the release stream once the manifest, partial upgrades, and the support window are all proven. Last, not first. ## Open questions @@ -658,6 +684,9 @@ Collected from `main` on 2026-07-27 (post-`v1.6.0`) and re-measured on 2026-08-2 | HelmReleases and pods moved by it | 23 of 95 HelmReleases, **43 of 164 pods replaced** | same run | | Pods restarted on a byte-identical image | Cilium 3, LINSTOR 8, MetalLB 4, objectstorage-controller 1 | same run; metallb pods came back on the digests both tags carry | | Artifacts bundling a library chart | 36 of 207, all `cozy-lib`, copied in by the `ArtifactGenerator` | `kubectl get ag -A -o yaml` | +| `cozy-lib` size and shape | 728 lines, 43 helpers; two of them (`rbac.subjectsForTenantAndAccessLevel` 27 consumers, `resources.defaultingSanitize` 21) carry 48 of ~55 consumer relationships | `packages/library/cozy-lib/templates/*.tpl` | +| `cozy-lib` with no consumer outside its own test chart | all of `_tls.tpl` (92 lines), 14 of the 15 helpers in `_cozyconfig.tpl`, and `rbac.subjectsForTenant` | same | +| The change that caused the 25-package fan-out | one added file, `_barman.tpl`, 24 lines, 2 consumers | `git diff --name-status v1.6.1..v1.6.2 -- packages/library/cozy-lib` | | source-watcher re-tar determinism | confirmed — 134 artifacts held a byte-identical digest across two pool revisions | same run | | Counterfactual: version removed from every vendored ref | 59 moved / 148 held, 16/95 HelmReleases, **21/164 pods** (against 73 / 134 / 23 / 43); the 14 eliminated are Cilium (6 artifacts), LINSTOR, linstor-gui, MetalLB, Multus, kubeovn-plunger, Kamaji, objectstorage-controller, seaweedfs-system | same stand, two rebuilt pools pushed and applied in sequence | | Data-plane pods restarted under that counterfactual | Cilium 0, LINSTOR 0, MetalLB 0, Multus 0, objectstorage-controller 0 | same run | From 7a4de7cbe05cbbd76048281e9887a2b4a5ebdddd Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Fri, 28 Aug 2026 12:11:15 +0500 Subject: [PATCH 12/13] docs(cozystack-as-a-distribution): answer the review, and name #43 as the split proposal Review asks for section 8 to be re-argued against a layered multi-repository split, on the premise that the document argues only against today's monorepo. The premise is half right and the earlier draft of this commit got the other half wrong by claiming the split has no written form. It does: #43 is the repository split, open since July, moving the application catalog out of the core repository into separately hosted, separately tested catalog repositories with three tiers assigned by explicit criteria. So section 8 now names #43 as the third position on its axis rather than implying the axis has two, and states the relationship plainly: a split decides where a package lives, this decides what its version means, and #43's own tier 1 and tier 2 catalogs need a version axis or each invents a local one. The scope table row for #43 said "component structure", which is what let it be read as something other than the split; it now leads with what it is. The increment the review actually adds over #43 is cutting the system layer out too. That part is unwritten, it is recorded as such in Alternatives, and it is not opposed: the version axis applies to it unchanged, requiresCore becoming "requires the system layer at or above" and the manifest pinning layer versions. One observation carried for whoever writes it. Coarser granularity does not remove the need for declared contracts, and the archive has a live example: _barman.tpl is an undeclared edge between apps/postgres and system/keycloak, keycloak being a consumer because it runs a CNPG Postgres of its own. Any split putting those two in different repositories makes that edge cross-repository rather than eliminating it. Three other things the review caught. The security-fix goal was wrong as written: under Option A the pool is republished, so the publication is still platform-wide and only the re-reconciliation is avoided. cozystack#3940 implements #18 and #23 and touches internal/operator/packagesource_reconciler.go, the same file the versioned pool changes, so it is in the scope table as a coordination point; its Tap is an aggregated-apiserver resource rather than a CRD, which puts the marketplace surface in core by this document's own boundary. And the cadence section now answers the e2e objection head on. Assisted-By: Claude Signed-off-by: Myasnikov Daniil --- .../cozystack-as-a-distribution/README.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/design-proposals/cozystack-as-a-distribution/README.md b/design-proposals/cozystack-as-a-distribution/README.md index 3d8d38a..e4e0523 100644 --- a/design-proposals/cozystack-as-a-distribution/README.md +++ b/design-proposals/cozystack-as-a-distribution/README.md @@ -23,10 +23,11 @@ This proposal is deliberately a joining piece. Several accepted or in-flight pro | Proposal | What it establishes | How this proposal relates | |---|---|---| -| [community#43](https://github.com/cozystack/community/pull/43) — Out-of-tree application catalogs (`@lllamnyp`, Review) | Three application tiers (platform / curated catalog / external catalog), catalog repositories as OCI artifacts, e2e cost as the driver, a declared minimum platform version checked at `PackageSource` reconciliation | **Supplies the version axis #43 assumes.** #43 defines the *component structure* of the distribution (Debian's main/universe split); this proposal defines what a version means inside it and where the declared minimum platform version comes from. #43 is the stronger near-term motivator and should not wait on this | +| [community#43](https://github.com/cozystack/community/pull/43) — Out-of-tree application catalogs (`@lllamnyp`, Review) | **The repository split.** Moves the application catalog out of the core repository into separately hosted, separately tested catalog repositories; three tiers (platform / curated catalog / external catalog) assigned by explicit criteria, catalog repositories as OCI artifacts, e2e cost as the driver, a declared minimum platform version checked at `PackageSource` reconciliation | **Supplies the version axis #43 assumes.** #43 defines the *component structure* of the distribution (Debian's main/universe split); this proposal defines what a version means inside it and where the declared minimum platform version comes from. #43 is the stronger near-term motivator and should not wait on this | | [community#18](https://github.com/cozystack/community/pull/18) — Cozymarketplace (`@kvaps`, **accepted**, merged 2026-08-24 as [`design-proposals/cozymarketplace`](https://github.com/cozystack/community/blob/main/design-proposals/cozymarketplace/README.md)) | Repository-as-unit, a krew-style meta-index, a repository is one versioned OCI artifact, `cozypkg` repository commands | **Adopted as-is for external repositories; extended, not replaced, for the platform's own.** #18 makes per-package pinning a non-goal **for its Phase 1**, which is the right call for the problem #18 is solving; it is a sequencing statement, not a permanent exclusion. This proposal is the Phase 2 it leaves room for, scoped to the first-party archive — see [§8](#8-repository-as-unit-versus-package-as-unit) | | [community#12](https://github.com/cozystack/community/pull/12) — Community package index and `cozypkg` authoring (`@kvaps`, `@IvanHunters`, closed 2026-07-16) | `cozypkg tap` / `init` / `push` / `search`, metadata-only index, `community.` name prefixing, an optional expected-signing-identity per entry | **Its package-level axis is revived here, on a different justification.** #12 was package-centric; #18 superseded it with a repository-centric model and #12 was closed. That supersession is sound on ecosystem grounds. This proposal re-derives the package-level axis from platform release engineering instead, and reuses #12's `cozypkg` surface and index-entry shape rather than inventing new ones | | [community#23](https://github.com/cozystack/community/pull/23) — Cozymarketplace supplementary (`@IvanHunters`, **accepted**, merged 2026-08-24 as [`design-proposals/cozymarketplace-supplementary`](https://github.com/cozystack/community/blob/main/design-proposals/cozymarketplace-supplementary/README.md)) | Marketplace endpoints in `cozystack-api`, a `TapIndex` cache, a pull credential set as `spec.secretRef` on the Flux source a tap already creates (no CRD change), `cozypkg validate` plus a two-lane index CI gate | **Consumed unchanged.** The `TapIndex` cache is the natural home for the release manifest / index reader described below. Note that #23 restates #18's position — "per-package version pinning remains out of scope, siding with `#18`" — so this proposal now differs from two accepted documents rather than one draft; see [§8](#8-repository-as-unit-versus-package-as-unit) | +| [cozystack#3940](https://github.com/cozystack/cozystack/pull/3940) — community marketplace implementation (`@IvanHunters`, open) | `cozypkg tap/untap/index/init/push/validate`, `Tap` as an aggregated-apiserver resource with a materializer, the packages-index and its publication gate | **Implements #18 and #23, and overlaps this proposal in one file.** Its tap materializer changes `internal/operator/packagesource_reconciler.go`, which is where the versioned pool's source-alias change also lands — worth coordinating rather than merging blind. Its `Tap` is served by `cozystack-api` rather than as a CRD, which places the marketplace surface in core by the boundary drawn in [§1](#1-four-tiers-one-package-model) | | [community#6](https://github.com/cozystack/community/pull/6) — ApplicationDefinition multi-version conversion (`@kvaps`, Draft) | `versions[]` with a storage version, `to`/`from` conversion templates, a `_version` stamp, a background migration controller | **Orthogonal axis, and the thing that makes semver honest.** #6 versions the *API surface* of an app; this proposal versions the *package*. Their relationship is what gives "breaking change" a testable definition — see [What MAJOR means](#what-major-means) | | [community#39](https://github.com/cozystack/community/pull/39) — Fold `extra` into `apps` (`@myasnikovdaniil`, Draft) | Retires the `extra` bucket; visibility / cardinality / protection / capability as declarative `ApplicationDefinition` fields | **Same direction, one tier down.** #39 removes a directory-as-metadata convention by moving the metadata onto the ApplicationDefinition; this proposal does the same for the version. Together they turn ApplicationDefinition into the package's control file | | [cozystack#3448](https://github.com/cozystack/cozystack/pull/3448) — ApplicationGroupDefinition (MVP) | Dynamic API-group registration, reserved-namespace validation, restart-based pickup via the `cozystack.io/config-hash` rollout | **Prerequisite for tier 2, and the proof that the runtime is already data-driven.** The mechanism it extends is the reason per-package delivery is feasible at all | @@ -110,7 +111,7 @@ None of those four require a marketplace, a community index, or a single externa - A Cozystack release is a **manifest** naming a core version and an exact version for every package and app it ships, and that manifest is a published artifact users and tooling can read. - Upgrading a release upgrades only the components whose versions changed. Unchanged components are not re-reconciled. - A package or app can ship a breaking change on its own major version without a platform major. -- A security fix in one package can be released and consumed without a platform release. +- A security fix in one package can be released and consumed without re-reconciling the platform. Note the precise claim: under Option A the pool artifact is republished, so the *publication* is still platform-wide; what the fix buys is that unchanged components keep their content and are not upgraded. Isolating the publication itself needs Option B. - "Breaking" has a written, CI-enforceable definition for a Cozystack package. - External catalogs (#43 tier 1/2) and community repositories (#18) are the same mechanism as first-party packages, differing only in origin and support level. - Release cadence is predictable one to two months ahead, with a published support window. @@ -473,6 +474,10 @@ This is the one place where this proposal could read as contradicting an accepte The extension proposed here comes from a requirement neither #12 nor #18 was scoped to weigh: **the platform's own release engineering.** #18's "tested together" argument is exactly the argument for a manifest — and a manifest that can only name whole repositories cannot express "these three packages moved and the other 155 did not". Shipping `2026.09` with the same core and three bumped apps is the entire value of the partial-upgrade goal, and it is unreachable if the finest addressable unit is the repository. So the divergence is narrow and does not touch #18's thesis: it is about whether the *first-party archive* is one repository or many packages, not about how community repositories should work. Nothing here asks #18 or #23 to be amended. +The axis has a third position, and it is already written down: **#43 is the repository split.** It moves the application catalog out of the core repository into separately hosted, separately tested catalog repositories, keeping in-tree only what the platform contract requires, with three tiers assigned by explicit criteria. This proposal is not opposed to it and says so twice — the scope table calls #43 the stronger near-term motivator, and [Alternatives](#alternatives-considered) records splitting first as compatible rather than rejected. The relationship is that a split decides *where a package lives* and this decides *what its version means*; #43's own tier 1 and tier 2 catalogs need a version axis or each will invent a local one. + +A further increment has been raised in review and is not yet written: cutting the system layer out too, so that core, system, and the app layers above it become separate repositories, each declaring a minimum version of the layer beneath. That is #43 plus one more cut. The version axis here applies to it unchanged — `requiresCore` becomes "requires the system layer at or above", and the manifest pins layer versions instead of package versions — so it is likelier complementary than opposed, on the same grounds as #43. + Both are true at different tiers, which is also how Debian works — the archive is versioned per package and resolved by a release; a third-party PPA is versioned as a unit and you take what it gives you. Two practical notes for whoever reconciles these. First, adopting per-package versioning for the first-party archive costs #18 nothing and asks nothing of it: its meta-index, tap flow, and repository-level tags are unaffected, and the manifest reader this proposal needs is the same `TapIndex` cache [#23](https://github.com/cozystack/community/pull/23) already specifies. Second, #12's concrete surface — `tap` / `untap` / `init` / `push` / `search`, `community.`-prefixed source names, metadata-only index entries with an optional expected signing identity — is reusable as written; the package-level axis is being revived here on new grounds, not the specific ergonomics being re-litigated. @@ -526,6 +531,7 @@ gantt Three rules make it hold: +0. **The cadence is not a new burden on e2e.** An objection worth answering directly: twelve trains a year, while e2e still fails on upstream dependencies, could mean more months skipped than shipped. The train adds no test that is not already run — it renames a rhythm the project already keeps, at 27, 34 and 30 days for the last three minors. What the objection does identify correctly is that skipping becomes *visible* where slipping was not, which is the point, and that the cure is landing the upgrade lane rather than avoiding the calendar. 1. **A train that is not ready skips the month; it never extends.** A calendar name makes slipping impossible to hide, and v1.6.0 needed four release candidates with a NO-GO on rc.1 for upgrade-only blockers. `2026.09` not existing is a clean, legible outcome. `2026.08` shipping on 12 September is not. 2. **Packages do not get release branches.** Only the train does. Packages are released from `main` and referenced by version in the manifest. Otherwise the branch topology multiplies by the number of packages. 3. **The support window is published before the cadence changes,** and the tooling has already chosen it. `.github/workflows/backport.yaml` enumerates the real `release-X.Y` branches, sorts them descending and targets the current and the previous one, so **the de-facto window today is two minor lines** — implemented, not merely intended. The proposal recommends ratifying that rather than inventing a different number: current minor plus the one before it, security-only for the older. Any other answer means changing the backport automation first, which is a cost that should be paid deliberately if at all. `supportedUpgradeFrom` in the manifest then becomes the machine-readable form of a promise the repository is already keeping. @@ -658,6 +664,7 @@ The ordering is chosen so that each phase is independently valuable and independ - **Per-package versions but no manifest — resolve with SemVer ranges at install time.** Rejected firmly. Ranges without a lockfile import npm's resolution problem into a platform with no lockfile and no ability to test the resolved set. The manifest *is* the lockfile. - **Let users compose versions freely.** Rejected as a supported mode. A distribution decides the set; that is what makes it testable and supportable. Holds exist as an escape hatch and are labelled unsupported. - **Split the repository first, version second** (i.e. do #43 before this). Not rejected — these are compatible and #43 has the more urgent driver in e2e cost. The note is only that #43's catalogs will need a version axis and will otherwise invent a local one per catalog. +- **Cut the system layer out as well**, so core, system and the app layers above it become separate repositories each declaring a minimum version of the layer beneath. This is #43 plus one more cut, raised in review and not yet written up; the increment over #43 is the system split, not the idea of splitting. Not evaluated here, and not opposed: the version axis applies to it unchanged, as it does to #43. One observation for whoever writes it. Coarser granularity does not by itself remove the need for declared contracts, and the archive has a live example: `_barman.tpl` is an undeclared edge between `apps/postgres` and `system/keycloak` ([§4](#4-the-versioned-pool)), keycloak being a consumer because it runs a CNPG Postgres of its own. Any split that puts those two in different repositories turns that edge into a cross-repository one rather than eliminating it, which is an argument for declaring contracts alongside a split rather than instead of one. - **Design a new migration framework from first principles.** Rejected in favour of generalising the four per-package hooks already in the tree. The existing pattern is proven, and its ordering guarantees are strictly better than the global counter's. ## Appendix A — measured facts From 7766bad5c8d616d62e3fd576772803cffa446cda Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Fri, 28 Aug 2026 13:01:45 +0500 Subject: [PATCH 13/13] docs(cozystack-as-a-distribution): the versioning unit is the PackageSource, and coarser breaks MAJOR Open question left standing: how do inter-package version dependencies work, and would versioning by group instead of by package avoid needing them. Both halves are now measured, and they point at a unit the document was not using. Group versioning does buy something. An edge needs a declared contract only when its two ends can move independently, so grouping absorbs most of them: at directory granularity 247 of the 251 dependsOn edges are intra-group and only 4 cross. But it costs the whole partial-upgrade goal. Every release touches every group, the last four transitions moved 6, 4, 5 and 6 of six, so "upgrade what changed" becomes "upgrade everything" again. The tree already carries a better unit. 31 of the 99 PackageSources span more than one top-level directory, 23 of them apps plus system, because an application's chart lives in apps/ while the ApplicationDefinition that serves it lives in system/-rd and neither installs without the other. All 251 dependsOn edges are source-to-source. That is a Debian source package producing several binaries at one version, already implemented. Section 3's manifest example was already keyed on source names before the prose caught up. Measured on v1.6.1 to v1.6.2 at that granularity: 49 of 99 sources hold today, 57 after the tag fix, and 83 after the tag fix and the library shrink. Against 0 of 5 for directory groups. So the unit keeps nearly all of the benefit at 99 objects rather than 164, and both things the proposal needs, a version and a dependency constraint, land on objects that already exist. Section 5 now carries the decisive objection to any coarser unit, which is not about convenience. Gate 2 verifies a declared MAJOR by diffing a generated OpenAPI schema; a group has no single surface to diff, so a group's MAJOR cannot be checked at all. #6's conversions are written between two storage versions of one app's API and have nowhere to attach on a group. And a group MAJOR tells an operator that something broke rather than what, dragging every unbroken package with it, so the signal decays from "you must act" to "somebody must act, possibly not you". Grouping stays a support and presentation concern, which is what section is for. Assisted-By: Claude Signed-off-by: Myasnikov Daniil --- .../cozystack-as-a-distribution/README.md | 36 +++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/design-proposals/cozystack-as-a-distribution/README.md b/design-proposals/cozystack-as-a-distribution/README.md index e4e0523..06d49a8 100644 --- a/design-proposals/cozystack-as-a-distribution/README.md +++ b/design-proposals/cozystack-as-a-distribution/README.md @@ -159,6 +159,25 @@ Eleven charts build their image from the repository root with `COPY api pkg cmd Everything else under `packages/{system,apps,extra}` is a package without argument. +#### What carries the version + +The tier says what a thing is. A separate question is what object the number attaches to, and the honest answer is not the chart directory. + +`PackageSource` already groups what has to ship together, and the tree proves it: 31 of the 99 sources span more than one top-level directory, 23 of them `apps` + `system`. That is not untidiness. An application's chart lives in `apps/` while the `ApplicationDefinition` that serves it lives in `system/-rd`, often with its operator, and none of the three is installable without the others. The dependency graph is drawn at the same granularity — all 251 `dependsOn` edges are source-to-source, not chart-to-chart. This is a Debian source package producing several binaries at one version, already implemented and already load-bearing. + +So the unit is the `PackageSource`: 99 of them rather than 164 charts. [§3](#3-the-release-manifest)'s manifest already keys on source names (`cozystack.linstor`, `cozystack.postgres-application`) — the example anticipated this before the prose did. Two things the proposal needs then land on objects that exist: a version on the source, and a constraint on the `dependsOn` edges it already declares. + +Measured on `v1.6.1` → `v1.6.2` at that granularity, and it is the number that decides the granularity question: + +| Unit | Held on a patch release | +|---|---| +| `PackageSource`, as things stand | 49 of 99 | +| `PackageSource`, after the tag fix | 57 of 99 | +| `PackageSource`, after the tag fix and the library shrink | **83 of 99** | +| Directory group (core / system / apps / extra / library) | **0 of 5** | + +The last row is not a rounding artefact. Every release touches every group: the four most recent transitions moved 6, 4, 5 and 6 groups out of the six that exist. + #### Two consequences the rest of this proposal depends on **The delivery boundary is in the wrong place today, but moving it needs no new machinery.** The installer chart renders exactly three things — the operator Deployment, the `cozy-system` namespace, and a pre-install Job that labels it. It carries no `crds/` directory and no CRD hook. What installs the CRDs depends on the path: the chart passes `--install-crds=true` and `--install-flux=true` to the operator (both default to `false` in the binary), so on the Helm path the operator installs them itself at startup from embedded manifests, via `crdinstall.Install` and `fluxinstall.Install`, then creates the platform `PackageSource`. On the `kubectl apply` path the same manifests ship as a release asset — `make manifests` concatenates `internal/crdinstall/manifests/*.yaml` into `_out/assets/cozystack-crds.yaml` and `hack/upload-assets.sh` uploads it beside the rendered operator — and the operator applies them idempotently afterwards. One source, two deliveries. @@ -406,6 +425,18 @@ The prior art for gate 2 is already in the tree and should be reused rather than Gate 2 is also where #6 and this proposal meet: #6 gives an app the ability to *survive* a storage-form change, and this proposal gives it the number that *advertises* one. A MAJOR without a `to`/`from` pair is a manual-action release; with one, it is transparent. +#### Why the unit cannot be coarser than an API surface + +This is the argument against versioning by group or by repository, and it is not about granularity of convenience. It is that a coarse unit makes MAJOR unverifiable and then meaningless. + +Gate 2 works by diffing a generated OpenAPI schema against the previous version of the same object. A `PackageSource` has one — its components' `ApplicationDefinition` schemas are a closed set. A group of thirty applications has no single API surface to diff, so there is nothing to compare a group's declared MAJOR against. The declaration stops being checkable, and an unchecked SemVer is a convention, which is the failure this section exists to prevent. + +The same applies to #6. A conversion is written between two storage versions of one app's API, and it lives with that app. A group MAJOR cannot carry `to`/`from` pairs, because there is no single schema being converted, so the mechanism that makes a breaking change survivable has nowhere to attach. + +And it degrades what the number tells an operator. A group MAJOR says something in the group broke, not what. Every package in the group that did not break is dragged to a major with it, so the signal moves from "you must act" to "somebody must act, possibly not you" — and after two releases where the major did not concern them, operators stop reading it. A version whose majors are usually irrelevant is worse than no version, because it costs attention and returns nothing. + +None of this argues against grouping. It argues that grouping is a support and presentation concern — the `section` field, the tiers in #43 — and not the thing the number attaches to. + ### 6. Migrations: per-package, generalising a pattern that already exists Today all 53 migrations live in `packages/core/platform/images/migrations/migrations/` as bare numbered shell scripts, are selected by `run-migrations.sh:39` walking `seq $CURRENT_VERSION $((TARGET_VERSION - 1))`, run as **cluster-admin** in one Job, and execute as a `pre-upgrade` hook on the `cozystack-platform` Helm release — which means *all* of them run before *any* component HelmRelease upgrades. The only state is `data.version` in the `cozy-system/cozystack-version` ConfigMap; there is no per-migration record, so every script must be idempotent, and that is convention rather than enforcement. @@ -664,6 +695,7 @@ The ordering is chosen so that each phase is independently valuable and independ - **Per-package versions but no manifest — resolve with SemVer ranges at install time.** Rejected firmly. Ranges without a lockfile import npm's resolution problem into a platform with no lockfile and no ability to test the resolved set. The manifest *is* the lockfile. - **Let users compose versions freely.** Rejected as a supported mode. A distribution decides the set; that is what makes it testable and supportable. Holds exist as an escape hatch and are labelled unsupported. - **Split the repository first, version second** (i.e. do #43 before this). Not rejected — these are compatible and #43 has the more urgent driver in e2e cost. The note is only that #43's catalogs will need a version axis and will otherwise invent a local one per catalog. +- **Version by group rather than by package** (one number for core, one for system, one per app layer), which is the shape a repository split implies. Rejected on measurement, and the measurement is one-sided. It does buy something real: an edge needs a declared contract only when its two ends can move independently, so grouping absorbs most of them — with directory-sized groups, 247 of the 251 `dependsOn` edges become intra-group and only 4 cross. But it costs the entire partial-upgrade goal. Every release touches every group; the last four transitions moved 6, 4, 5 and 6 of the six directory groups, so "upgrade what changed" degenerates to "upgrade everything". Against that, versioning the `PackageSource` holds 83 of 99 units on the same release ([§1](#1-four-tiers-one-package-model)). Trading a 15-declaration problem for the whole benefit is not a good trade. The decisive objection is separate and is in [§5](#5-what-major-means): a group has no API surface to diff, so a group's MAJOR cannot be verified, cannot carry #6's conversions, and cannot tell an operator which component actually broke. - **Cut the system layer out as well**, so core, system and the app layers above it become separate repositories each declaring a minimum version of the layer beneath. This is #43 plus one more cut, raised in review and not yet written up; the increment over #43 is the system split, not the idea of splitting. Not evaluated here, and not opposed: the version axis applies to it unchanged, as it does to #43. One observation for whoever writes it. Coarser granularity does not by itself remove the need for declared contracts, and the archive has a live example: `_barman.tpl` is an undeclared edge between `apps/postgres` and `system/keycloak` ([§4](#4-the-versioned-pool)), keycloak being a consumer because it runs a CNPG Postgres of its own. Any split that puts those two in different repositories turns that edge into a cross-repository one rather than eliminating it, which is an argument for declaring contracts alongside a split rather than instead of one. - **Design a new migration framework from first principles.** Rejected in favour of generalising the four per-package hooks already in the tree. The existing pattern is proven, and its ordering guarantees are strictly better than the global counter's. @@ -694,6 +726,10 @@ Collected from `main` on 2026-07-27 (post-`v1.6.0`) and re-measured on 2026-08-2 | `cozy-lib` size and shape | 728 lines, 43 helpers; two of them (`rbac.subjectsForTenantAndAccessLevel` 27 consumers, `resources.defaultingSanitize` 21) carry 48 of ~55 consumer relationships | `packages/library/cozy-lib/templates/*.tpl` | | `cozy-lib` with no consumer outside its own test chart | all of `_tls.tpl` (92 lines), 14 of the 15 helpers in `_cozyconfig.tpl`, and `rbac.subjectsForTenant` | same | | The change that caused the 25-package fan-out | one added file, `_barman.tpl`, 24 lines, 2 consumers | `git diff --name-status v1.6.1..v1.6.2 -- packages/library/cozy-lib` | +| `PackageSource` objects, and how many span directories | 99 total, 31 span more than one (23 `apps`+`system`, 7 `extra`+`system`, 1 all three) | `packages/core/platform/sources/*.yaml` | +| `dependsOn` edges, and how many cross a directory group | 251 total, 247 within, **4 crossing** | same | +| Held on a patch release, by candidate unit | `PackageSource` 49 of 99 today, 57 after the tag fix, **83 after the tag fix and the library shrink**; directory group **0 of 5** | `v1.6.1..v1.6.2` against the source definitions | +| Groups touched per release | 6, 4, 5 and 6 of six, over the last four transitions | `git diff --name-only .. -- packages/` | | source-watcher re-tar determinism | confirmed — 134 artifacts held a byte-identical digest across two pool revisions | same run | | Counterfactual: version removed from every vendored ref | 59 moved / 148 held, 16/95 HelmReleases, **21/164 pods** (against 73 / 134 / 23 / 43); the 14 eliminated are Cilium (6 artifacts), LINSTOR, linstor-gui, MetalLB, Multus, kubeovn-plunger, Kamaji, objectstorage-controller, seaweedfs-system | same stand, two rebuilt pools pushed and applied in sequence | | Data-plane pods restarted under that counterfactual | Cilium 0, LINSTOR 0, MetalLB 0, Multus 0, objectstorage-controller 0 | same run |