Chromium Density & QPS Benchmarks - #6898
Conversation
95239c9 to
8893ac8
Compare
|
/cc @Nishanth29 |
|
|
||
| successful = result.get("successful_sessions", 0) | ||
| failed = result.get("failed_sessions", 0) | ||
| agg = result.get("aggregate", {}) |
There was a problem hiding this comment.
If the API returns aggregate as null on an error, reading metrics below might throw a None error. Can we do result.get("aggregate") or {} instead?
| ns = FLAGS.k8s_agentic_namespace | ||
| logging.info("Cleanup: deleting SandboxClaims and draining warm pool.") | ||
|
|
||
| # Delete any lingering SandboxClaims to release claimed pods |
There was a problem hiding this comment.
In Cleanup(), should we delete by label instead of --all so other workloads in the namespace don't get deleted?
There was a problem hiding this comment.
Good Catch, thanks. updated to use -l _WARMPOOL_LABEL so it strictly only deletes the Chromium claims generated by this benchmark.
| _emit(samples, aggregate, "claim_p95_ms", "claim_p95", "ms", ns, extra) | ||
|
|
||
| # Throughput and counts | ||
| samples.append( |
There was a problem hiding this comment.
What does "samples" mean here? It seems to be more like "results". Please document.
There was a problem hiding this comment.
Samples is the list of perfkitbenchmarker.sample.Sample objects that PKB expects the Run() function to return. aggregate is the JSON dictionary of metrics calculated and returned by our FastAPI agent. Added inline documentation for these.
| mode = FLAGS.k8s_qps_mode | ||
|
|
||
| if mode == "raw_claim": | ||
| return _RunRawClaim(benchmark_spec) |
There was a problem hiding this comment.
Is this "raw_claim" mode used & documented?
There was a problem hiding this comment.
Yes, it is. It is used to bypass the FastAPI agent entirely and fire SandboxClaims directly via kubectl to test the raw throughput of the Kubernetes API server and the SandboxController. I've added a brief inline comment here as well for clarity.
| samples = [] | ||
|
|
||
| # TTFE latency stats | ||
| _emit(samples, aggregate, "ttfe_mean_ms", "ttfe_mean", "ms", ns, extra) |
There was a problem hiding this comment.
What are "samples" and "aggregate"? Please document.
There was a problem hiding this comment.
Samples is the list of perfkitbenchmarker.sample.Sample objects that PKB expects the Run() function to return. aggregate is the JSON dictionary of metrics calculated and returned by our FastAPI agent. Added inline documentation for these.
| # TTFE latency stats (computed from raw claim results) | ||
| if ttfe_values: | ||
| n = len(ttfe_values) | ||
| samples.append( |
There was a problem hiding this comment.
Why don't you use _emit here and below?
There was a problem hiding this comment.
This is just due to an architectural difference between the two modes.
In agent mode, the FastAPI agent calculates all the percentiles and returns them in the aggregate dictionary, which our utility function parses. In raw_claim mode, we bypass the agent completely. The Python script collects the raw timestamps and calculates the percentiles locally using the _percentile() helper. Because we don't have an aggregate dictionary from the API, we append the sample.Sample objects directly.
| ns, | ||
| extra, | ||
| ) | ||
| ) |
There was a problem hiding this comment.
See my comments in the previous PR. Please simplify the code by leverage reusable utility methods.
3 participants
Files: 2 new files
perfkitbenchmarker/linux_benchmarks/kubernetes/agentic/k8s_chromium_density_benchmark.pyperfkitbenchmarker/linux_benchmarks/kubernetes/agentic/k8s_qps_benchmark.pyDescription: Adds two more GKE Agent Sandbox benchmarks:
Chromium Density: Measures browser interaction latency (navigate, evaluate, click, fill, screenshot) under concurrent Chromium sandbox sessions. Uses CDP (Chrome DevTools Protocol) from the orchestrator — no Node.js in the sandbox.
QPS Saturation: Measures scheduling throughput by firing sandbox claims at controlled QPS rates. Supports two modes:
agent(via orchestrator API) andraw_claim(direct SandboxClaim creation via kubectl). Identifies the QPS saturation point where warm pool drain causes TTFE spikes.