Skip to content

🌱 Ensure COS phase immutability for referenced object approach#2635

Open
pedjak wants to merge 1 commit intooperator-framework:mainfrom
pedjak:secret-verify
Open

🌱 Ensure COS phase immutability for referenced object approach#2635
pedjak wants to merge 1 commit intooperator-framework:mainfrom
pedjak:secret-verify

Conversation

@pedjak
Copy link
Copy Markdown
Contributor

@pedjak pedjak commented Apr 8, 2026

Description

ClusterObjectSet phases are immutable by design, but when objects are stored in
external Secrets via refs, the Secret content could be changed by deleting and
recreating the Secret with the same name. This enforces phase immutability for
the referenced object approach by:

  • Verifying that referenced Secrets have immutable: true set
  • Computing a per-phase SHA-256 content digest after successful phase resolution
    and recording it in .status.observedPhases
  • Blocking reconciliation (Progressing=False, Reason=Blocked) if any referenced
    Secret is mutable or any phase's resolved content digest has changed

The digest is source-agnostic — it covers the fully resolved phase content
regardless of whether objects are inline or referenced from Secrets, making it
forward-compatible with future object sources (e.g., bundle refs).

Blocked COS resources are recoverable: they can be re-reconciled when triggered
(e.g., via annotation), re-evaluating the condition.

Reviewer Checklist

  • API Go Documentation
  • Tests: Unit Tests (and E2E Tests, if appropriate)
  • Comprehensive Commit Messages
  • Links to related GitHub Issue(s)

Copilot AI review requested due to automatic review settings April 8, 2026 15:28
@netlify
Copy link
Copy Markdown

netlify bot commented Apr 8, 2026

Deploy Preview for olmv1 ready!

Name Link
🔨 Latest commit 2f24934
🔍 Latest deploy log https://app.netlify.com/projects/olmv1/deploys/69da50baa10a0c00099cbb52
😎 Deploy Preview https://deploy-preview-2635--olmv1.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@openshift-ci openshift-ci bot requested review from joelanford and oceanc80 April 8, 2026 15:28
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Enforces ClusterObjectSet phase immutability when objects are sourced via referenced Secrets by requiring referenced Secrets to be immutable and by detecting Secret content changes using recorded hashes.

Changes:

  • Add Secret verification in the COS reconciler (immutable requirement + SHA-256 hash comparison) and block reconciliation when verification fails.
  • Persist referenced Secret hashes in .status.observedObjectContainers and extend CRD/applyconfigurations accordingly.
  • Add/extend unit + e2e coverage and update docs to reflect the new behavior.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
test/e2e/steps/steps.go Adds new e2e steps for triggering reconciliation, message fragment matching, and checking observed Secret hashes in status.
test/e2e/features/revision.feature Adds e2e scenarios for “mutable Secret” and “recreated Secret with changed content” blocking behavior.
manifests/experimental.yaml Extends CRD schema with .status.observedObjectContainers.
manifests/experimental-e2e.yaml Extends e2e CRD schema with .status.observedObjectContainers.
internal/operator-controller/controllers/resolve_ref_test.go Updates ref-resolution tests to use immutable Secrets.
internal/operator-controller/controllers/clusterobjectset_controller_internal_test.go Adds unit tests for Secret hashing and referenced-Secret verification.
internal/operator-controller/controllers/clusterobjectset_controller.go Implements referenced Secret verification + hashing and blocks reconciliation on violations.
helm/olmv1/base/operator-controller/crd/experimental/olm.operatorframework.io_clusterobjectsets.yaml Mirrors CRD schema changes for Helm packaging.
docs/draft/concepts/large-bundle-support.md Updates documented behavior/conventions for referenced Secrets (immutability + hash enforcement).
applyconfigurations/utils.go Registers apply-configuration kind for ObservedObjectContainer.
applyconfigurations/api/v1/observedobjectcontainer.go Adds generated apply configuration for the new status type.
applyconfigurations/api/v1/clusterobjectsetstatus.go Adds apply support for .status.observedObjectContainers.
api/v1/zz_generated.deepcopy.go Adds deepcopy support for ObservedObjectContainer and status field.
api/v1/clusterobjectset_types.go Introduces ObservedObjectContainer API type and status field.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +605 to +614
func ClusterObjectSetReportsConditionWithMessageFragment(ctx context.Context, revisionName, conditionType, conditionStatus, conditionReason string, msgFragment *godog.DocString) error {
msgCmp := alwaysMatch
if msgFragment != nil {
expectedMsgFragment := substituteScenarioVars(strings.Join(strings.Fields(msgFragment.Content), " "), scenarioCtx(ctx))
msgCmp = func(actualMsg string) bool {
return strings.Contains(actualMsg, expectedMsgFragment)
}
}
return waitForCondition(ctx, "clusterobjectset", substituteScenarioVars(revisionName, scenarioCtx(ctx)), conditionType, conditionStatus, &conditionReason, msgCmp)
}
Copy link

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This step normalizes whitespace in the expected fragment but not in the actual condition message; if the controller formats messages with newlines/multiple spaces (common when wrapping), strings.Contains may fail unexpectedly. Consider normalizing actualMsg with the same strings.Fields + Join approach before Contains to reduce e2e flakiness while still matching by fragment.

Copilot uses AI. Check for mistakes.
@pedjak pedjak changed the title ✨ Ensure COS phase immutability for referenced object approach 🌱 Ensure COS phase immutability for referenced object approach Apr 8, 2026
Copilot AI review requested due to automatic review settings April 8, 2026 15:46
@pedjak pedjak requested review from camilamacedo86 and dtfranz and removed request for oceanc80 April 8, 2026 15:54
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

internal/operator-controller/controllers/clusterobjectset_controller.go:1

  • Treating Progressing=False, Reason=Blocked as terminal at the predicate level can prevent the controller from ever reconciling again, even if the user fixes the root cause (e.g., recreates the Secret as immutable, or changes COS spec/generation). If “Blocked” is intended to be recoverable, remove it from this predicate or gate suppression more narrowly (e.g., only suppress when generation hasn’t changed, while still allowing spec updates / explicit triggers to enqueue reconciles).
//go:build !standard

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@codecov
Copy link
Copy Markdown

codecov bot commented Apr 8, 2026

Codecov Report

❌ Patch coverage is 71.17117% with 32 lines in your changes missing coverage. Please review.
✅ Project coverage is 68.94%. Comparing base (dd57c28) to head (2f24934).

Files with missing lines Patch % Lines
api/v1/zz_generated.deepcopy.go 33.33% 8 Missing ⚠️
applyconfigurations/api/v1/observedphase.go 0.00% 8 Missing ⚠️
...troller/controllers/clusterobjectset_controller.go 90.36% 6 Missing and 2 partials ⚠️
...plyconfigurations/api/v1/clusterobjectsetstatus.go 0.00% 6 Missing ⚠️
applyconfigurations/utils.go 0.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2635      +/-   ##
==========================================
+ Coverage   68.92%   68.94%   +0.02%     
==========================================
  Files         140      141       +1     
  Lines        9905    10005     +100     
==========================================
+ Hits         6827     6898      +71     
- Misses       2566     2593      +27     
- Partials      512      514       +2     
Flag Coverage Δ
e2e 37.41% <0.00%> (-0.41%) ⬇️
experimental-e2e 52.49% <65.13%> (+0.09%) ⬆️
unit 53.65% <60.36%> (+0.05%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

// +listType=map
// +listMapKey=name
// +optional
ObservedObjectContainers []ObservedObjectContainer `json:"observedObjectContainers,omitempty"`
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of requiring the individual objects to never change (and keeping track of per-object hashes), WDYT about computing a single hash of the resulting phases content. If that remains stable, do we care if the underlying organization of the secrets changed?

If we do that, it also means we abstract away any details of how we ultimately end up with those phases (i.e. we don't care if it is inline vs from secrets vs from a bundle ref, etc.), which would be more futureproof.

Copy link
Copy Markdown
Contributor Author

@pedjak pedjak Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we do that, it also means we abstract away any details of how we ultimately end up with those phases (i.e. we don't care if it is inline vs from secrets vs from a bundle ref, etc.), which would be more futureproof.

interesting idea! implemented, ptal.


func (c *ClusterObjectSetReconciler) SetupWithManager(mgr ctrl.Manager) error {
skipProgressDeadlineExceededPredicate := predicate.Funcs{
skipTerminallyBlockedPredicate := predicate.Funcs{
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A terminally blocked COS could become unblocked if the user puts back the expected content though, right? We should continue reconciling to check if the expected content is back in place.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — removed ClusterObjectSetReasonBlocked from the skip predicate so blocked COS can be re-reconciled. If the original content is restored, the digest check passes and the COS resumes normal operation. Added an e2e recovery scenario to verify this.

@pedjak pedjak requested a review from joelanford April 9, 2026 07:56
// object content at first successful reconciliation.
//
// +required
Hash string `json:"hash"`
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By creating the resolution digest we still needing this one: https://github.com/operator-framework/operator-controller/pull/2610/changes#diff-c804d15cab34d4c7c30fcd68d40d0269084c11db0736d844bd6ee8f60262a924R68

If so, could we change the name PhaseDigest?

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 7 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +161 to +169
currentPhases := make([]ocv1.ObservedPhase, 0, len(phases))
for _, phase := range phases {
hash, err := computePhaseHash(phase)
if err != nil {
setRetryingConditions(cos, err.Error())
return ctrl.Result{}, fmt.Errorf("computing phase hash: %v", err)
}
currentPhases = append(currentPhases, ocv1.ObservedPhase{Name: phase.GetName(), Hash: hash})
}
Copy link

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The phase hash is computed from the boxcutter phases returned by buildBoxcutterPhases(), after the reconciler mutates each resolved object (e.g., injecting labels.OwnerNameKey). This couples the stored .status.observedPhases hashes to controller-internal mutations and COS metadata, so future controller changes (or label changes) could falsely appear as “referenced content changed” and permanently block reconciliation. Consider hashing the pre-mutation resolved manifests (or normalizing/stripping controller-added fields) so the hash reflects only the referenced object content you’re trying to make immutable.

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to this comment. We should hash what the user provides.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — digest is now computed before controller mutations (label injection, collision protection). The hash reflects only the user-provided resolved content.

Copilot AI review requested due to automatic review settings April 9, 2026 16:05
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 15 out of 15 changed files in this pull request and generated 1 comment.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +171 to +177
if len(cos.Status.ObservedPhases) == 0 {
cos.Status.ObservedPhases = currentPhases
} else if err := verifyObservedPhases(cos.Status.ObservedPhases, currentPhases); err != nil {
l.Error(err, "resolved phases content changed, blocking reconciliation")
markAsNotProgressing(cos, ocv1.ClusterObjectSetReasonBlocked, err.Error())
return ctrl.Result{}, nil
}
Copy link

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ObservedPhases is only populated when empty (line 172), but is never updated to include newly added phases. This breaks the immutability guarantee for phases added after initial reconciliation. When a new phase is added, it bypasses verification (line 796 checks only stored phases), and subsequent content changes to that phase won't be detected because it was never added to ObservedPhases. After verification passes, any new phases in currentPhases should be appended to cos.Status.ObservedPhases so they're tracked for future tamper detection.

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a valid concern. The COS spec (including phases) is immutable — enforced by a CEL validation rule that prevents spec changes after creation. Phases can never be "added after initial reconciliation." The set of phases is fixed at COS creation time, so ObservedPhases will always capture all phases on the first successful resolution, and subsequent reconciliations will always have the same set of phases to verify against.

@pedjak pedjak requested a review from camilamacedo86 April 9, 2026 22:25
Copilot AI review requested due to automatic review settings April 9, 2026 22:45
Copy link
Copy Markdown
Contributor

@camilamacedo86 camilamacedo86 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems fine for me 🎉
Thank you for looking on that.
/approve

@openshift-ci
Copy link
Copy Markdown

openshift-ci bot commented Apr 10, 2026

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: camilamacedo86

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 10, 2026
// name is the phase name matching a phase in spec.phases.
//
// +required
Name string `json:"name"`
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Include validation markers to match spec phase name?

	// +kubebuilder:validation:MinLength=1
	// +kubebuilder:validation:MaxLength=63
	// +kubebuilder:validation:XValidation:rule=`!format.dns1123Label().validate(self).hasValue()`,message="the value must consist of only lowercase alphanumeric characters and hyphens, and must start with an alphabetic character and end with an alphanumeric character."

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — added MinLength, MaxLength, and DNS1123 label validation markers matching spec phase name.

// object content at first successful resolution.
//
// +required
Digest string `json:"digest"`
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Include validation markers that reject invalid digest strings? For futureproofing, should we make the format of this string like the following?

<digestAlgoritm>:<digest>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — digest now uses <algorithm>:<hex> format (e.g., sha256:abcdef...). Added validation marker: self.matches('^[a-z0-9]+:[a-f0-9]+$').

// +listType=map
// +listMapKey=name
// +optional
ObservedPhases []ObservedPhase `json:"observedPhases,omitempty"`
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar validations for the spec phases list?

	// +kubebuilder:validation:XValidation:rule="self == oldSelf || oldSelf.size() == 0", message="observedPhases is immutable"
	// +kubebuilder:validation:MaxItems=20

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — added immutability CEL rule (self == oldSelf || oldSelf.size() == 0) and MaxItems=20.

Comment on lines +372 to +375
if cnd := meta.FindStatusCondition(rev.Status.Conditions, ocv1.ClusterObjectSetTypeProgressing); cnd != nil && cnd.Status == metav1.ConditionFalse {
if cnd.Reason == ocv1.ReasonProgressDeadlineExceeded {
return false
}
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before and after are exactly equivalent here, right? If so, revert this change to avoid the extra conditional nesting?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted — flattened back to a single condition.

func computePhaseDigest(phase boxcutter.Phase) (string, error) {
h := sha256.New()
fmt.Fprintf(h, "phase:%s\n", phase.GetName())
for _, obj := range phase.GetObjects() {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Double checking: phase.GetObjects() returns the list of objects in a deterministic order?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes — GetObjects() returns objects in the same order as specPhase.Objects, which is an immutable spec slice. The order is deterministic across reconciliations.

// produces a canonical encoding with sorted map keys.
func computePhaseDigest(phase boxcutter.Phase) (string, error) {
h := sha256.New()
fmt.Fprintf(h, "phase:%s\n", phase.GetName())
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WDYT about this:

phaseMap := map[string]any{
  "name": phase.GetName(),
  "objects": phase.GetObjects(),
}
phaseData, err := json.Marshal(phaseMap)
// handleErr

h.Write(phaseData)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — switched to single json.Marshal of a map[string]any{"name": ..., "objects": ...}.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — adopted this approach. computePhaseDigest now marshals a map with name + objects into a single JSON blob before hashing.

Comment on lines +747 to +752
if prev, ok := storedMap[c.Name]; ok && prev != c.Digest {
return fmt.Errorf(
"resolved content of phase %q has changed (expected digest %s, got %s): "+
"a referenced object source may have been deleted and recreated with different content",
c.Name, prev, c.Digest)
}
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: collect all the diffs and report all of them?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — verifyObservedPhases now collects all mismatched phases and reports them in a single error message.

Comment on lines +793 to +795
if secret.Immutable == nil || !*secret.Immutable {
return fmt.Errorf("secret %s/%s is not immutable: referenced secrets must have immutable set to true", ref.namespace, ref.name)
}
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same nit: collect all mutable secrets and report all of them?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — verifyReferencedSecretsImmutable now collects all mutable secrets and reports them in a single error message.

}

t.Run("deterministic for same content", func(t *testing.T) {
phase := makePhase("deploy", makeObj("v1", "ConfigMap", "cm1"))
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Put 100 different objects into the phase and compute the digest 100 times and make sure it's always the same?

Seems like that would cover my earlier question about deterministic ordering of the objects with a very high probability.

Also maybe a different test with the same objects but provided in a different order, which would prove that the input order matches the output order, which would cause the digest to change?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — added a "deterministic with many objects" test that creates 100 ConfigMaps, computes the digest 100 times, and asserts they're all identical. Also added a "different order produces different digest" test.

Comment on lines +320 to +327
t.Run("passes when current has new phase not in stored", func(t *testing.T) {
stored := []ocv1.ObservedPhase{{Name: "deploy", Digest: "sha256:abc123"}}
current := []ocv1.ObservedPhase{
{Name: "deploy", Digest: "sha256:abc123"},
{Name: "crds", Digest: "sha256:def456"},
}
assert.NoError(t, verifyObservedPhases(stored, current))
})
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In theory, this should never happen, right? But if it does happen shouldn't this return an error?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — verifyObservedPhases now returns an error when stored is empty. Also added a length check (stored != current count) to catch phase count changes.

require.NoError(t, err)
})

t.Run("deduplicates refs to same secret", func(t *testing.T) {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: the test doesn't seem like it proves anything about deduplicating refs to the same secret.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — renamed to "checks secret only once when referenced multiple times".

And resource "deployment/test-httpd" is installed
And ClusterObjectSet "${COS_NAME}" has observed phase "resources" with a non-empty digest

Scenario: ClusterObjectSet blocks reconciliation when referenced Secret is not immutable
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: double negative

Suggested change
Scenario: ClusterObjectSet blocks reconciliation when referenced Secret is not immutable
Scenario: ClusterObjectSet blocks reconciliation when referenced Secret is mutable

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — scenario title is now "ClusterObjectSet blocks reconciliation when referenced Secret is mutable".

secret ${TEST_NAMESPACE}/${COS_NAME}-mutable-secret is not immutable: referenced secrets must have immutable set to true
"""

Scenario: ClusterObjectSet blocks reconciliation when referenced Secret content changes
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This (or another) scenario that verifies that we will start reconciling again if the original content is put back and the hashes match once again?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — added recovery steps to the content change scenario: after the COS is blocked, we restore the original secret content, trigger reconciliation, and verify the COS resumes with Progressing=True/Succeeded.

// verifyObservedPhases compares current per-phase digests against stored
// digests. Returns an error naming the first mismatched phase.
func verifyObservedPhases(stored, current []ocv1.ObservedPhase) error {
storedMap := make(map[string]string, len(stored))
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of checks we should add for doing the pairwise comparison:

if len(stored) == 0 {
	return nil
}
if len(stored) != len(current) {
	return fmt.Errorf("number of phases has changed (expected %d phases, got %d)", len(stored), len(current))
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — verifyObservedPhases now checks: (1) stored is not empty, (2) len(stored) == len(current), (3) all digests match. Tests cover all three cases.

.bingo/gojq.mod Outdated
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something wrong with the rebase? We removed gojq in a different commit recently, right?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, Claude messed things up... fixed now.

ClusterObjectSet phases are immutable by design, but when objects are
stored in external Secrets via refs, the Secret content could be changed
by deleting and recreating the Secret. This enforces phase immutability
by:

- Verifying that referenced Secrets have `immutable: true` set
- Computing a per-phase SHA-256 content digest of pre-mutation resolved
  objects and recording it in `.status.observedPhases`
- Blocking reconciliation (`Progressing=False, Reason=Blocked`) if any
  referenced Secret is mutable or any phase's digest has changed
- Allowing blocked COS to recover when original content is restored

The digest is source-agnostic — it covers fully resolved phase content
regardless of whether objects are inline or from Secrets, making it
forward-compatible with future object sources.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api-diff-lint-override approved Indicates a PR has been approved by an approver from all required OWNERS files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants