Warmpool & Deletion Benchmarks - #6899
Conversation
f66a61f to
a524471
Compare
|
/cc @pmandewalkar |
|
|
||
| # Drain to 0 for clean measurement (moved from Prepare for sweep compatibility) | ||
| utils.DrainWarmPool(ns, warmpool_name, label, timeout=int(drain_timeout)) | ||
| time.sleep(2) |
There was a problem hiding this comment.
Saw these a couple times in the test, could we maybe use a more deterministic method to avoid waiting too long/too little if the system resolves earlier/later? Maybe a polling loop or kubectl command checking the state of what we are waiting for.
There was a problem hiding this comment.
The utils.DrainWarmPool actually is a deterministic polling loop (it polls kubectl get pods until the count is exactly 0). The time.sleep(2) that follows it is just a brief buffer to ensure the Kubernetes API cache has fully settled before we start the timer for the next phase. I've added an inline comment to clarify this.
| logging.info("Provisioning... %d/%d (%.0f%%)", running, batch_size, pct) | ||
| if running >= batch_size: | ||
| break | ||
| time.sleep(3) |
There was a problem hiding this comment.
Since we have a configurable poll_interval that is used during the deletion loop on line 258, should we replace this hardcoded time.sleep(3) with time.sleep(poll_interval) for consistency?
There was a problem hiding this comment.
yeah That was a hardcoded oversight, thanks for catching this. I've updated it to use time.sleep(poll_interval) for consistency.
| # Pods we never saw disappear (stuck) get the full drain time | ||
| for pn in pod_names_before: | ||
| if pn not in pod_gone_times: | ||
| pod_gone_times[pn] = total_drain_time |
There was a problem hiding this comment.
Is there a way we are differentiating pods in the edge case that take the whole drain time to successfully delete vs pods that are stuck and are assigned the same drain time? Maybe we could track a count of stuck pods and push it through as an additional metric (like k8s_deletion_stuck_pods_count) so we can separate that?
|
|
||
| ip_after = _CountAllocatedIPs(ns, label) | ||
| deletion_rate = ( | ||
| (len(pod_names_before) / total_drain_time) if total_drain_time > 0 else 0 |
There was a problem hiding this comment.
If pods are left stuck, when we use len(pod_names_before) will it assume all pods were successfully deleted and inflate the deletion_rate? Since we are already tracking the exact number of deleted pods inside the loop on line 244, could we use the deleted variable as the numerator here?
There was a problem hiding this comment.
Modified this so we now track stuck_pods_count during the loop. The deletion rate is now calculated strictly against actual_deleted = len(pod_names_before) - stuck_pods_count.
I've added a new metric gke_deletion_stuck_pods_count to the emitted samples so we can track this edge case if needed somewhere downstream.
| provision_start = time.monotonic() | ||
| _PatchReplicas(ns, warmpool_name, batch_size) | ||
|
|
||
| deadline = time.time() + provision_timeout |
There was a problem hiding this comment.
Would it be better to stick to time.monotonic() rather than using and comparing different measures? Since time.time() could be affected by system clock adjustments, it could theoretically make the timeout weird.
There was a problem hiding this comment.
updates so that both to use time.monotonic().
| deadline = t_scale + threshold_s | ||
| first_pod_time = None | ||
|
|
||
| while time.time() < deadline: |
There was a problem hiding this comment.
Same question as before regarding monotonic() vs time()
There was a problem hiding this comment.
updates so that both to use time.monotonic().
pmandewalkar
left a comment
pmandewalkar
left a comment
There was a problem hiding this comment.
Left a few comments regarding clock and hardcoded sleeps.
a524471 to
9b1e3cc
Compare
| } | ||
|
|
||
|
|
||
| def _EmitLifecycleSamples(samples: list, lifecycle: dict, namespace: str, extra: dict) -> None: |
There was a problem hiding this comment.
Please document what's "lifecycle", and what's the difference between _emit and _EmitLifecycleSamples
3 participants
Files: 2 new files
perfkitbenchmarker/linux_benchmarks/kubernetes/agentic/k8s_warmpool_benchmark.pyperfkitbenchmarker/linux_benchmarks/kubernetes/agentic/k8s_deletion_benchmark.pyDescription: Adds two infrastructure lifecycle benchmarks:
Warmpool Scale-Up: Measures how quickly N sandbox pods can be provisioned from zero via the SandboxWarmPool controller. Scrapes pod lifecycle timestamps (creationTimestamp, PodScheduled condition, Ready condition) for time-to-created/scheduled/running percentiles. No agent API needed — interacts directly with the Kubernetes API.
Deletion & Cleanup: Measures bulk deletion efficiency and IP address reclamation. Provisions N pods via SandboxWarmPool, bulk-deletes them by scaling to zero, and tracks per-pod deletion latency and IPAM release timing. Tests whether "Terminating" pods hold IP addresses longer than expected.