Feature: ReadYourWritesConsistency

[alvaroaleman]

Upstream is discussing to mark the RV offficially as comparable: kubernetes/enhancements#5505
If/When that happens, we could extend the cache-backed client to provide read your own write semantics, for example by:

• Blocking in mutating operations until the RV in the cache is >= response RV
• Blocking all reading operations in the cache until it it got to >= response RV

Tasks

• v0.25
  • Design: ✨ Read your own write client design #3473
  • Initial implementation: ✨ Add a read-your-own-write client #3472
• v0.25.1
  • 🐛 client: fix subresource create RV parse error under read-your-writes consistency #3591
• Follow-ups: (more to be added once we merged the design)
  • Optionally use Applyconfigurations: Generate GetResourceVersion kubernetes/kubernetes#141688 (once merged)
  • Consider evaluating shouldByPassCache and only wait if the client would use the cache

[sbueringer]

In general makes sense to me.

I think we should definitely have an option to turn it off though for better performance.

Blocking in mutating operations until the RV in the cache is >= response RV

I think this one absolutely makes sense

Blocking all reading operations in the cache until it it got to >= response RV

I"m not so sure about this one, I assume this would essentially block all reads for a kind if there is a single write for that kind.
I think this could slow down controllers considerably. Especially if there are a lot of reconcilers that are using the same objects.

[alvaroaleman]

Delete will be tricky due to #3059

[sbueringer]

Not sure I follow. Can't we wait until the object is gone from the cache?

(just have to be careful for the case where another object with the same name is created right after deletion)

EDIT: Ah, do you mean the case where a finalizer is set or not? I would hope we can handle both cases

1. => wait until object is gone or
2. => wait until resourceVersion is high enough

[alvaroaleman]

Yeah, because of finalizers. The problem is that since we don't deserialize the object back if it isn't gone from storage, we don't know the post-delete RV, so 2) in your list doesn't have an obvious solution

[sbueringer]

Got it. If we can't come up with a good user-facing API for Delete we could overwrite Delete in the cached client and handle the response differently there?

[serathius]

Hey everyone, I look at KEP-5505 in a little different perspective, would love to hear your opinion on this. My background is in etcd and API server cache performance/correctness, so I'm newer to the intricacies of controller-runtime.

For me, KEP-5505 is a huge deal. It addresses the complexity of eventual consistency caused by watch. A tradeoff Kubernetes made for scale that put a burden on controller authors. It unlocks the biggest advantage of etcd: the totally ordered revision for every operation. This is the same "magic" that powered the recent API server cache snapshotting improvements (https://kubernetes.io/blog/2025/09/09/kubernetes-v1-34-snapshottable-api-server-cache/,), and now it can be used to build more reliable controllers.

I've seen production incidents where this complexity leads to failure: controllers falling minutes behind, resulting in a DaemonSet creating duplicate pods or a scheduler placing pods on nodes without capacity. It's an unreasonable burden to expect every developer to perfectly handle unbounded staleness, especially when it's only discovered during an incident.

Because of this, I strongly agree with @alvaroaleman's idea of Blocking all reading operations in the cache until it got to >= response RV even make it a default guard rail. As @sbueringer mentioned, it should absolutely be disable-able for performance-critical scenarios, but the default should be safety over performance.

To address the performance concerns, perhaps the API could be more flexible than a simple on/off switch. What if a controller could specify an acceptable staleness? We could also expose informer delay as a metric (like in etcd-io/etcd#20260), allowing controllers to automatically abort reconciliation if staleness exceeds a critical threshold.

I'm not sure what the best user-facing API is or where it should live (informer, controller-runtime, etc.). That's why I'm hoping to combine my perspective with your controller expertise here. Thanks!

[sbueringer]

I think the following could also be an interesting idea:

• Provide a guarantee that a read call will see at least all previous writes done by the same controller/binary/client
• If the cache catches up quickly enough (configurable "timeout"), serve from cache
• Otherwise fallback to live call

(I don't remember exactly, but I think this would be similar to how one/some of the apiserver caches are working?)

[serathius]

Provide a guarantee that a read call will see at least all previous writes done by the same controller/binary/client

Right, that has been available in API as Not older than request, but not in informer (https://kubernetes.io/docs/reference/using-api/api-concepts/#semantics-for-get-and-list).

If the cache catches up quickly enough (configurable "timeout"), serve from cache

This is what we do for consistent read in watch cache, where we wait for cache to catch up to etcd. Default timeout is 3 seconds. https://kubernetes.io/blog/2024/08/15/consistent-read-from-cache-beta/. Consistent read only differs in that instead of waiting for RV from previous write, it asks etcd what is the latest RV. Either of should be good.

Otherwise fallback to live call

I would refrain from live calls for LIST if possible. They don't have a bounded cost, even with limit as object might differ a lot in size causing high decoding cost. The only exception would be Gets on single object. API server never serves Gets from cache, only from etcd as a way to reduce lock contention. Might be something we consider to avoid issues like kubernetes/kubernetes#130767.

[halfcrazy]

I've been exploring a similar approach and would like to share my prototype implementation for feedback.

Approach: ConsistentClient Wrapper

Instead of modifying the existing Client interface directly, I implemented a ConsistentClient wrapper that provides additional methods with cache consistency guarantees:

type ConsistentClient interface {
    Client
    CreateConsistent(ctx context.Context, obj Object, opts ...CreateOption) error
    UpdateConsistent(ctx context.Context, obj Object, opts ...UpdateOption) error
    PatchConsistent(ctx context.Context, obj Object, opts ...PatchOption) error
    DeleteConsistent(ctx context.Context, obj Object, opts ...DeleteOption) error
    StatusConsistent() ConsistentSubResourceWriter
}

Usage:

consistentClient := client.NewConsistentClient(mgr.GetClient(), mgr.GetCache())
err := consistentClient.CreateConsistent(ctx, obj)
// After this returns, cache is guaranteed to be synced

Implementation Details

The implementation polls the cache until synchronization is confirmed. The key challenge is correctly identifying when the cache is "up-to-date", especially in edge cases.

For Create/Update/Patch (waitForCacheSync)

poll:
  ├── NotFound → keep waiting
  │
  ├── UID differs:
  │     ├── cache.CreationTimestamp > target → success (cache has newer object)
  │     └── cache.CreationTimestamp <= target → keep waiting (stale data)
  │
  └── UID matches && RV >= target → success

// waitForCacheSync polls the cache until the object with matching UID has
// a ResourceVersion greater than or equal to the target ResourceVersion.
//
// Note: If the object is deleted after the write operation and no new object
// is created, this will keep polling until the context is cancelled.
// Callers should use an appropriate context timeout.
func (c *consistentClient) waitForCacheSync(ctx context.Context, obj Object) error {
	targetRV := obj.GetResourceVersion()
	targetUID := obj.GetUID()
	targetCreationTimestamp := obj.GetCreationTimestamp()
	key := ObjectKeyFromObject(obj)

	return wait.PollUntilContextCancel(ctx, c.pollInterval, true, func(ctx context.Context) (bool, error) {
		current := obj.DeepCopyObject().(Object)
		if err := c.cacheReader.Get(ctx, key, current); err != nil {
			if apierrors.IsNotFound(err) {
				// Object not yet in cache (or was deleted), keep polling.
				// We cannot distinguish between "not yet synced" and "deleted",
				// so we rely on context timeout for the deleted case.
				return false, nil
			}
			return false, err
		}

		// Check UID to ensure we're looking at the same object.
		if current.GetUID() != targetUID {
			// UID differs - cache has a different object with the same name.
			// Check CreationTimestamp to determine if we should keep waiting:
			// - If cache object is newer (created after ours), cache is up-to-date
			//   with the latest state (our object was deleted and replaced).
			// - If cache object is older, it's stale data not yet removed - keep waiting.
			if current.GetCreationTimestamp().After(targetCreationTimestamp.Time) {
				return true, nil
			}
			return false, nil
		}

		// Compare ResourceVersion to check if cache is synced
		cmp, err := resourceversion.CompareResourceVersion(current.GetResourceVersion(), targetRV)
		if err != nil {
			// If comparison fails, continue polling
			return false, nil
		}
		return cmp >= 0, nil
	})
}

For Delete (waitForCacheDelete)

poll:
  ├── NotFound → success
  │
  ├── UID differs:
  │     ├── cache.CreationTimestamp > target → success (object replaced)
  │     └── cache.CreationTimestamp <= target → keep waiting (stale data)
  │
  └── UID matches → keep waiting

// waitForCacheDelete polls the cache until the object with the specified UID
// is no longer present.
//
// Success conditions:
// - Object not found in cache (NotFound error)
// - Object exists but has different UID and newer CreationTimestamp (original was deleted, new one created)
func (c *consistentClient) waitForCacheDelete(ctx context.Context, obj Object, uid types.UID, creationTimestamp metav1.Time) error {
	key := ObjectKeyFromObject(obj)

	return wait.PollUntilContextCancel(ctx, c.pollInterval, true, func(ctx context.Context) (bool, error) {
		current := obj.DeepCopyObject().(Object)
		err := c.cacheReader.Get(ctx, key, current)
		if apierrors.IsNotFound(err) {
			// Object is gone from cache - success
			return true, nil
		}
		if err != nil {
			return false, err
		}

		// If UID is different, check CreationTimestamp to determine cache state
		if current.GetUID() != uid {
			// If cache object is newer, original was deleted and replaced - success
			// If cache object is older, it's stale data - keep waiting
			if current.GetCreationTimestamp().After(creationTimestamp.Time) {
				return true, nil
			}
			return false, nil
		}

		// Same object still exists in cache, keep polling
		return false, nil
	})
}

Why UID + CreationTimestamp?

Comparing only ResourceVersion is insufficient for these edge cases:

1. Object deleted and recreated with same name: The new object will have a different UID and potentially lower RV. Without UID check, we might incorrectly wait forever or match the wrong object.

2. Cache contains stale object: When UID differs, we need CreationTimestamp to distinguish between:

   • Cache has an older object (stale, not yet removed) → keep waiting
   • Cache has a newer object (our object was deleted and replaced) → cache is up-to-date

Important: Context Timeout Behavior

In certain edge cases, the "keep waiting" state will continue polling until context cancellation/timeout:

1. Create/Update/Patch: If the object is deleted immediately after write and no new object is created, cache returns NotFound indefinitely. We cannot distinguish between "not yet synced" and "actually deleted".

2. Delete: If the cache has a stale object with the same UID that never gets cleared (unlikely but theoretically possible), polling continues.

This means callers MUST use an appropriate context timeout:

ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
err := consistentClient.CreateConsistent(ctx, obj)
if errors.Is(err, context.DeadlineExceeded) {
    // Handle timeout - object may have been deleted after creation
}

This is a deliberate design choice - we prefer explicit timeout handling over potentially incorrect "success" responses. However, I'd like to discuss if there are better approaches for these edge cases.

Trade-offs

Pros:

• Non-invasive: Doesn't change existing Client interface
• Opt-in: Users choose when they need consistency guarantees
• Uses official k8s.io/apimachinery/pkg/util/resourceversion.CompareResourceVersion for RV comparison

Cons:

• Polling-based (vs event-driven) - though with small intervals (10ms default), this is negligible
• Requires passing both Client and Cache reader to constructor
• Relies on context timeout for certain edge cases (see above)

Questions for Discussion

1. Would this approach be acceptable for inclusion in controller-runtime, or would you prefer modifying the core Client interface?

2. Should we consider an "overlay cache" approach (as described in Java Operator SDK blog) for immediate consistency without polling?

3. Any concerns about the UID + CreationTimestamp logic for edge case handling?

4. Block on write vs block on read?

   My current implementation blocks on the mutating operation (CUD). As you mentioned in the issue, an alternative is to block subsequent read operations. Trade-offs:

   Approach	Pros	Cons
   Block on CUD	Simpler mental model; write returns = cache synced	Increases write latency; caller may not need immediate read
   Block on Get	Write returns faster; only pays cost when needed	More complex state tracking; need to track "pending" RVs per object

   Which approach do you think fits better with controller-runtime's design philosophy?

Happy to share the full implementation or submit a PR if there's interest!

[alvaroaleman]

@halfcrazy if we add something to controller-runtime itself, I want it to be push- and not poll-based so the perf impact is minimal

[csviri]

Hi All, just putting here our 2 cents, in Java Operator SDK we solve this with an additional cache on top of Informer's cache - (calling this it read-cache-after-write consistency ) - see details here:
operator-framework/java-operator-sdk#2944
So basically a push based approach. We have a poll based utility also, but that will be deprecated after this is released.

Just one note, since would be interested also on your opinion: as far we can see this is not just about cache, so what we do in the current implementation is that we also filter out the event for our change (or up to our change). Thus if we did the change on a resource, and we are now sure that the most recent resource is in the cache, we don't trigger the reconiliation for our own change (thus for the event for that change), only if someone else is changes the resource (not our controller).
I think this is desirable in the general case.
What do you think?

(Of course, users have the option always to reschedule reconciliation explicitly)

Note also that handling delete is an issue properly is an issue with the current Kubernetes API delete response:
operator-framework/java-operator-sdk#3117

thank you!