Pod Snapshot Benchmark - #6900
Conversation
ffbd559 to
18c8521
Compare
|
/cc @pmandewalkar |
| pool.submit(_ApplyClaim, sname, namespace, step_template) | ||
|
|
||
| logging.info("Waiting for %d source pod(s) Running + preload", burst_size) | ||
| with ThreadPoolExecutor(max_workers=workers) as pool: |
There was a problem hiding this comment.
Would it be better to batch the polling to see the status of all pods at once (we could maybe use label selectors). I'm worried that 100 subprocesses could be an issue.
There was a problem hiding this comment.
The Snapshot benchmark tests heavy I/O state hydration, The burst_size defaults to 1, and in our saturation sweeps, we only pushed it to a maximum of 5. This can be more of a concern for very high-scale tests.
18c8521 to
2b8ab93
Compare
roycaihw
left a comment
roycaihw
left a comment
There was a problem hiding this comment.
It's hard to follow what this benchmark is doing with many unclear variable names / abbreviations and a long script. I think this one needs better documentation and potentially splitting the huge script into smaller meaningful methods
| preload_mode: str, | ||
| ) -> bool: | ||
| """Render a SandboxTemplate that runs a user-provided startup script.""" | ||
| script_path = preload_mode.split(":", 1)[1] |
There was a problem hiding this comment.
Please document the schema of preload_mode
There was a problem hiding this comment.
Done. schema expects script:<absolute_path_to_script>.
| ], | ||
| "nodeSelector": _get_sandbox_node_selector(), | ||
| "tolerations": _get_sandbox_tolerations(), | ||
| "restartPolicy": "OnFailure", |
There was a problem hiding this comment.
Shall we move the yaml template into separate template files? I see you doing that in other places and PRs. Shall we be consistent?
There was a problem hiding this comment.
I actually kept this one inline using json.dumps() because are dynamically injecting a multi-line bash script (read from the workload scripts) into the container's args array, and JSON serialization handles complex string escaping and newlines automatically and safely. YAML Jinja templating can be highly prone to indentation breaks and escaping errors.
|
|
||
| logging.info("Waiting for %d source pod(s) Running + preload", burst_size) | ||
| with ThreadPoolExecutor(max_workers=workers) as pool: | ||
| source_futs = [ |
There was a problem hiding this comment.
Was shorthand for futures (from the ThreadPoolExecutor). Renamed all instances of futs to futures.
| ) -> dict: | ||
| """Execute one full snapshot/restore cycle and return a result dict. | ||
|
|
||
| Handles source creation, snapshot, restore, TTFE measurement, |
There was a problem hiding this comment.
It refers to the original pod that boots up and runs the preload script, whose state is then captured by the snapshot. I've updated the docstring for _RunSnapshotCycle to explicitly define this.
3 participants
Files: 1 new file
perfkitbenchmarker/linux_benchmarks/kubernetes/agentic/k8s_snapshot_benchmark.pyDescription: Adds the GKE Pod Snapshot saturation benchmark:
Pod Snapshot: Measures GKE Pod Snapshot create/restore latency under varying memory preload sizes. Creates source pods with configurable memory footprints, triggers CRIU checkpoints to GCS-backed storage, restores from snapshots, and measures TTFE (Time To First Execution) for restored pods. Includes:
--skip_snapshotmode for cold-start TTFE measurement (no snapshot/restore)--preload_mode=script:<path>for custom startup scripts (vibe coding simulation)DeploySnapshots()integration for GCS bucket, managed folder, KSA, and IAM setupThis is the only benchmark that uses GKE Pod Snapshots (CRIU + GCS), PodSnapshotManualTrigger CRDs, and creates its own SandboxTemplates per measurement (rather than using the shared warm pool).