Subscription pointing at a copied CSV becomes permanently orphaned after the original AllNamespaces installation is removed

[mkolesnik]

Bug Report

What did you do?

1. Installed an AllNamespaces operator (sailoperator) in the operators namespace using the pre-existing global-operators OperatorGroup.
2. OLM copied the CSV into all namespaces.
3. Created a second namespace (test-sail) with its own OperatorGroup and a Subscription for the same package.
4. OLM accepted the second Subscription, set status.installedCSV and status.currentCSV, and created an InstallPlan with phase: Complete.
   However, the CSV in test-sail was still the copied CSV (olm.copiedFrom: operators, status.reason: Copied), and no deployment was created in test-sail.
5. Deleted the original Subscription and CSV from the operators namespace.
6. OLM garbage-collected all copied CSVs cluster-wide, including the one in test-sail.

# 1. Install AllNamespaces in operators namespace
kubectl apply -f - <<'EOF'
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: sailoperator
  namespace: operators
spec:
  channel: stable
  name: sailoperator
  source: operatorhubio-catalog
  sourceNamespace: olm
  installPlanApproval: Automatic
EOF

# 2. Wait for the CSV to install
CSV=$(kubectl get sub sailoperator -n operators -o jsonpath='{.status.currentCSV}' --timeout=60s)
kubectl wait csv "$CSV" -n operators --for=jsonpath='{.status.phase}'=Succeeded --timeout=120s

# 3. Verify OLM copied the CSV into other namespaces
kubectl get csv "$CSV" -n default -o jsonpath='{.metadata.labels.olm\.copiedFrom}'
# operators

# 4. Create second Subscription in a different namespace
kubectl create ns test-sail
kubectl apply -f - <<'EOF'
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: test-sail-og
  namespace: test-sail
spec:
  targetNamespaces: []
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: sailoperator
  namespace: test-sail
spec:
  channel: stable
  name: sailoperator
  source: operatorhubio-catalog
  sourceNamespace: olm
  installPlanApproval: Automatic
EOF

# 5. Verify OLM piggybacked on the copied CSV
kubectl get sub sailoperator -n test-sail -o jsonpath='{.status.installedCSV}'
# (same CSV name as above)
kubectl get csv "$CSV" -n test-sail -o jsonpath='{.metadata.labels.olm\.copiedFrom}'
# operators
kubectl get deployments -n test-sail
# (empty -- no real installation)

# 6. Remove original installation
kubectl delete sub sailoperator -n operators
kubectl delete csv "$CSV" -n operators

# 7. Observe: Subscription is orphaned
kubectl get csv -A | grep sail
# (empty -- all CSVs gone)
kubectl get sub sailoperator -n test-sail -o jsonpath='state={.status.state} installedCSV={.status.installedCSV}'
# state=UpgradePending installedCSV=<stale CSV name>
kubectl get deployments -n test-sail
# (empty)

What did you expect to see?

One of:

1. OLM reconciles the test-sail Subscription after the copied CSV is garbage-collected and creates a real installation (CSV + deployment) in test-sail.
2. OLM does not set status.installedCSV on a Subscription that points at a copied CSV it does not own, and instead creates a real installation in test-sail from the start.
3. OLM reports an error condition on the Subscription indicating the operator is already installed via AllNamespaces in another namespace.

What did you see instead? Under which circumstances?

The test-sail Subscription is permanently orphaned.
It reports installedCSV pointing at a CSV that does not exist anywhere on the cluster.
No error condition is set on the Subscription.
The only condition is CatalogSourcesUnhealthy: AllCatalogSourcesHealthy.

The catalog-operator logs show "skipping update: installplan already created" and "all subscriptions up to date" because the InstallPlan in test-sail is already phase: Complete and approved: true.
The InstallPlan also references the same CSV, which was the copied CSV from the operators namespace, not a real installation in test-sail.

Restarting both olm-operator and catalog-operator does not fix it.
Annotating the Subscription to trigger a reconciliation does not fix it.

Note: this also reproduces without the AllNamespaces scenario.
Installing an operator in a single namespace, then deleting only the CSV, leaves the Subscription in the same orphaned state.
This was reported in #1441 (April 2020) and closed as stale despite a commenter confirming it was still reproducible on OLM 0.16.1.

Environment

• operator-lifecycle-manager version: 0.43.0 (quay.io/operator-framework/olm@sha256:cbe85a690175f6ae2c140aa8a0e3fbe1a5ffe4cdd2d82191003cd353f71395f3). Changelogs for 0.44.0 and 0.45.0 do not include relevant fixes.
• Kubernetes version: Server v1.35.0, Client v1.31.1
• Kubernetes cluster kind: Kind (Kubernetes in Docker)

Possible Solution

The catalog-operator should detect that a Subscription's status.installedCSV references a CSV that no longer exists and trigger a new resolution cycle, regardless of whether the InstallPlan is already complete.

In the AllNamespaces case specifically, OLM should either not set status.installedCSV when the CSV is a copy (olm.copiedFrom label present), or it should detect the loss of the copied CSV and reconcile.

Additional context

The root cause appears to be that once an InstallPlan reaches phase: Complete, the catalog-operator considers the Subscription fully satisfied and does not re-evaluate it, even when the CSV it references is deleted.
In the AllNamespaces variant, this is made worse by OLM pointing the Subscription at a copied CSV that the Subscription has no ownership over, so the CSV can be removed without OLM noticing.