Design: decompose ModelCache into a per-cluster ModelCacheHydration by dennis-upbound · Pull Request #362 · modelplaneai/modelplane

dennis-upbound · 2026-07-21T16:55:17Z

Description of your changes

A design doc, not code yet — opening as a draft for direction before implementation.

ModelCache runs every matched cluster's hydration lifecycle inline in one fan-out function (PVC + Job + token Secret + a four-phase machine + drop-after-Ready + token resolution, threading cluster_name through every method), and it's the one fan-out that doesn't follow ModelDeployment → ModelReplica. Separately (#186), a cache's clusterSelector and a deployment's placement are decided independently, so a replica can land on a cluster the cache never staged to and fail to mount at runtime, with nothing visible at apply time.

This proposes handling both together: a per-cluster ModelCacheHydration child (the cache analogue of ModelReplica) that owns one cluster's hydration, and making the ModelCache footprint follow where its referencing replicas are placed rather than a hand-maintained selector. Covers #210 and #186. Includes an architecture diagram and a lifecycle flow diagram.

Where I'd most like judgment (all in the doc): the unified vs phased delivery, keeping modelCacheRef vs deriving the model from the deployment, the fleet-reconciler-vs-reuse-ModelCache ownership question, and the child kind name (ModelCacheHydration vs ModelCacheReplica).

design/modelcache-hydration.md only; no code.

I have:

Read and followed Modelplane's contribution process.
~~Run nix flake check (or ./nix.sh flake check) and made sure it passes.~~ (design doc only, no code paths)
~~Added or updated tests covering any composition function changes.~~ (design doc only)
Signed off every commit with git commit -s.

ModelCache runs every matched cluster's hydration lifecycle inline in one fan-out function, and its footprint (clusterSelector) can silently diverge from where replicas are scheduled, so a replica can land on a cluster the cache never staged to and fail to mount at runtime. This design proposes a per-cluster ModelCacheHydration child, mirroring ModelReplica, and makes the cache footprint follow placement. It covers issues modelplaneai#210 and modelplaneai#186, with architecture and lifecycle diagrams. Signed-off-by: Dennis Ramdass <dennis@upbound.io>

haarchri · 2026-07-21T19:35:28Z

+- **Resolve** the referencing `ModelReplica`s (the placed set) and, for pre-warm,
+  the `clusterSelector`.
+- **Stamp** one child per cluster in `placed ∪ preWarm`, named
+  `child_name("modelcache", ns, cache-name, cluster)`, copying down `huggingFace`,


do we have here a problem with 63 char limit ?

No 63-char problem here: resource.child_name is DNS-label-safe by construction. It joins the parts, truncates the prefix, and appends a 5-char hash, so the result is always a valid label at or under 63 chars, including the child's own name with cluster folded in. The PVC/Job/Secret keep the cluster out of their names (child_name("modelcache", ns, cache-name[, "hydrate"|"auth"])) so they match the serving-side mount, and those stay ≤63 too.

Added a line to the naming section spelling this out. Pushed.

haarchri · 2026-07-21T19:44:30Z

+
+### Hydrating before ready
+
+A replica can be scheduled onto a cluster new to the model before its PVC exists.


If hydration on that cluster fails permanently, expired HF token, storage exhausted, bad revision, the replica appears to be gated indefinitely.
This needs a backoff plus a Failed condition that propagates up to the ModelDeployment, otherwise a credential problem on one cluster surfaces as a deployment that simply never becomes ready?!

Good catch, this is the real gap in the gate. It had no failure exit: a bad token or bad revision would hold the replica in Hydrating forever.

The child already has a Failed phase, so the fix is to use it: a failed hydration surfaces as ArtifactReady=False with reason HydrationFailed on the ModelCache, and the gated replica fails with that reason instead of gating indefinitely. The hydration Job's backoffLimit bounds retries, so a permanent failure stops and is reported rather than retried forever.

Added a failure path to Hydrating before ready. Pushed.

The hydration gate had no failure exit: a bad token or bad revision would hold the replica in Hydrating forever. Add a Failed path that surfaces ArtifactReady=False and fails the replica, bounded by the Job's backoffLimit. Note that child_name keeps every composed name a valid 63-char DNS label. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Dennis Ramdass <dennis@upbound.io>

negz

I like this direction overall. The two things that need the most thought before we proceed:

Does this need to wait for crossplane/crossplane#7572?
Is hydrate-on-demand definitely desirable?

negz · 2026-07-21T19:36:49Z

+1. requires every `ModelReplica` in the fleet (the scheduler already does this for
+   capacity accounting);
+2. keeps those that reference it (`spec.modelCacheRef.name` and namespace);


We could make this more efficient by:

Have ModelDeployment label replicas with the cache they use (modelplane.io/cache: foo) when it creates them

Have ModelCache require only replicas with that label.

With pre-warm the parent doesn't walk replicas at all. The footprint is the clusterSelector, so compose-model-cache fans out on the selector and never requires the replica set. The label is a nice optimization for the on-demand path, but that path is now the rejected alternative.

negz · 2026-07-21T19:40:29Z

+2. **Derive the footprint from placement.** `ModelCache` stages onto the
+   clusters where its referencing replicas are placed, not onto a hand-maintained
+   `clusterSelector`. A selector, if set, is an opt-in pre-warm set, not the whole
+   footprint.


I suggest we rename or move the selector to make it clear that it doesn't actually limit where the cache is placed. e.g. spec.preWarm.clusterSelector or similar.

Under pre-warm the selector is the footprint, so it keeps clusterSelector and its plain meaning: it's exactly where the cache is placed. The "doesn't limit placement" ambiguity came from the placement-derived model, which is gone.

negz · 2026-07-21T19:47:14Z

+The status quo, and the divergence #186 is about: the selector and placement
+drift, and the failure is a runtime mount error with nothing at apply time.
+Deriving the footprint from placement removes the class of bug by construction.
+The selector survives only for the case placement can't serve, pre-warming ahead
+of a deployment.


This isn't exactly the status quo since #189.

Corrected. #189 is now called out explicitly as an alternative ("Constrain placement to the footprint"): it constrains MD placement to the cache footprint, and this proposal relaxes that and loads from source off the footprint instead.

negz · 2026-07-21T19:49:04Z

+That is a larger, user-facing change with its own migration. This proposal keeps
+`modelCacheRef` and solves the divergence, waste, sprawl, and lifecycle problems
+without it. Identity-keyed sharing can be added on top later, since the child
+already carries a source independent of how it was requested.


I'm not advocating for this alternative at all, but: this is a non-issue at this stage in our lifecycle.

If there is a better design, we should adopt it now. Even if it requires breaking API changes and "migrations". No one is running Modelplane v0.2 - there's no one to break, no one to migrate.

Agreed, and that's the spirit of the rework: went straight to pre-warm rather than keeping the hedged both-modes design. Kept modelCacheRef only because it's load-bearing for the cache-vs-source decision, not for migration reasons.

negz · 2026-07-21T20:01:33Z

+The selector survives only for the case placement can't serve, pre-warming ahead
+of a deployment.
+
+### Drop `modelCacheRef`; derive the model from the deployment


What is the purpose of ModelCache? Is it:

Get weights on a cluster before any model replica runs there

Get weights on a cluster the first time a model replica runs there

A bit of both?

Apart from storage, 1. feels optimal to me. I'm basing this on my experience trying to run larger models like Kimi when testing. It was something like 45 minutes to hydrate the cache PVC, then another 15+ minutes to load from the cache PVC.

With 1., the 45 mins is paid in advance. So the user experience for an MD author is "my model is ready in ~15 mins".

With 2., the 45 mins is paid the first time the model is deployed on the cluster. So for that MD author the user experience is "my model is ready in an hour", because we need to do two copies. We hydrate the cache, then load the model to the replica(s). Subsequent deploys are faster.

I think this alternative would be 2., whereas the actual proposal is 3 (a bit of both).

If it were possible I'd prefer optimizing for 1.:

Drop modelCacheRef

Place the cache on all clusters that match its selector

Schedule MDs to any cluster that matches the MD's selector - regardless of whether there's a cache there

In this world it's up to the platform team to make sure the cache is where they want it. A model can't implicitly carry it to a new cluster the platform team didn't pre-hydrate. No MD ever pays the "double download" tax. They either load from cache, or load directly from the source.

All that said, I can't think of a way to actually make 1. work given that the MD needs to know whether its loading from cache or not when forming its engine args.

Went with option 1. On the engine-args blocker: compose-model-deployment already resolves the cache footprint (resolve_cache_footprint, the #189 path), so it knows per-replica whether that replica's cluster is in the footprint. It passes that down, and base.cache_mount injects --model=<mount> on footprint clusters and leaves the engine loading from source elsewhere. The decision is known at compose time, so forming args needs no runtime lookup, so option 1 works, and it drops the #7572 dependency too. New "Cache or source, per replica" section spells this out.

injects --model=<mount>

Doesn't this go against our overarching design principle of not injecting engine args though? What if a new engine shipped tomorrow that used a different flag for this?

I know we already inject a bit today, but I was hopeful that was something temporary we could walk back.

Thinking out loud, some options we could consider:

Admit we need to inject flags, transparent passthrough of engines isn't realistic.

Loosen to allow injecting flags, but only when they're best-effort optimizations (like using a cache).

Allow the user to specify conditional args - e.g. change these args if there's a cache.

negz · 2026-07-21T20:21:10Z

+A replica can be scheduled onto a cluster new to the model before its PVC exists.
+Rather than fail the mount, the replica gates readiness on its cluster's
+`ModelCacheHydration`: it reports `Hydrating` until the PVC is Bound, then
+proceeds. The first replica on a new cluster pays the download once, as a visible
+state rather than a mount error.


Can you clarify: is the ModelReplica watching the ModelCacheHydration to know when it's ready? Or the PVC directly? Watching the MCH seems best (e.g. if one day we have caches that don't use PVCs).

The ModelCacheHydration object, not the PVC, for exactly your reason (a future non-PVC cache keeps the same readiness contract). Stated explicitly now in "Hydrating before ready."

negz · 2026-07-21T22:14:50Z

+2. On upgrade the child adopts the existing, already-Bound PVC instead of
+   provisioning a new one and re-downloading the weights.
+
+## Placement-driven footprint


I think this'll need crossplane/crossplane#7572 to function correctly. Without that functionality nothing will trigger the ModelCache function to run when new ModelReplicas are placed, so the ModelCache function won't know to create a new ModelCacheHydration.

Given crossplane/crossplane#7572 is approved I expect it'll ship next month in Crossplane v2.4. We may want to hold this implementation until then.

This was the strongest reason to move off placement-derived. Pre-warm removes the dependency: the footprint is the static clusterSelector, so ModelCache doesn't need a watch on ModelReplica placement to stay correct. No hold needed. #7572 is now noted as required only by the on-demand alternative.

Adopt Nic's direction: the cache's clusterSelector is the authoritative footprint the platform team pre-warms, rather than deriving the footprint from replica placement. A deployment loads from the cache where staged and from the source elsewhere, decided at compose time via the footprint resolution that already exists (resolve_cache_footprint). This removes the double-download tax and the crossplane#7572 dependency, and relaxes the modelplaneai#189 placement constraint. Keep the ModelCacheHydration decomposition. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Dennis Ramdass <dennis@upbound.io>

dennis-upbound · 2026-07-22T16:29:27Z

Went with pre-warm on both counts.

crossplane#7572: only the placement-derived footprint needs it. Pre-warm makes clusterSelector authoritative, so ModelCache never watches replica placement and needs no cross-resource trigger. No hold on v2.4.
hydrate-on-demand: dropped as the default. It's now the rejected alternative (the double-download tax, the #7572 dependency, and a model carrying private weights onto clusters the platform team didn't choose). Pre-warm pays the big hydration once, ahead of deployment; a deployment loads from the cache where it's staged and from the source elsewhere.

Doc reworked around this. Inline replies below.

Add an explicit approval line so a review means something clear: it covers the ModelCacheHydration decomposition and the pre-warm-authoritative footprint, including load-from-source off the footprint. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Dennis Ramdass <dennis@upbound.io>

vercel Bot deployed to Preview – modelplane-docs July 21, 2026 16:56 View deployment

dennis-upbound force-pushed the design/modelcache-hydration branch from bf7fe2c to 8830612 Compare July 21, 2026 17:00

vercel Bot deployed to Preview – modelplane-docs July 21, 2026 17:01 View deployment

dennis-upbound force-pushed the design/modelcache-hydration branch from 8830612 to 4feb19b Compare July 21, 2026 17:15

vercel Bot deployed to Preview – modelplane-docs July 21, 2026 17:16 View deployment

haarchri reviewed Jul 21, 2026

View reviewed changes

dennis-upbound requested a review from negz July 21, 2026 20:22

negz reviewed Jul 21, 2026

View reviewed changes

vercel Bot deployed to Preview – modelplane-docs July 28, 2026 13:29 View deployment


		### Hydrating before ready

		A replica can be scheduled onto a cluster new to the model before its PVC exists.

Uh oh!

Conversation

dennis-upbound commented Jul 21, 2026

Description of your changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

negz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dennis-upbound commented Jul 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants