feat(agentx): aggregate full-response ITL and interactivity - #2504
feat(agentx): aggregate full-response ITL and interactivity#2504cquil11 wants to merge 5 commits into
Conversation
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Beyond the inline finding, I checked two other candidate issues in the new aggregation and ruled both out: stats dropping when request start/end ns metadata is missing, and degenerate zero/non-positive values not being filtered — neither holds up against the actual aggregation logic.
Extended reasoning...
Two additional candidate issues around the new full_response_output_token_throughput_per_user aggregation were examined this run and ruled out: (1) concern that records missing request_start_ns/request_end_ns metadata would silently drop the new stat — this is not the case, extract_per_record_floats/stats_for operate independently of the start/end-ns filtering used for duration, so the new metric's per-record extraction is unaffected by that gate; (2) concern that degenerate zero/non-positive full_response_output_token_throughput_per_user values would skew mean/percentiles unfiltered — to_float only excludes None/non-numeric values, but this mirrors how input_tokens/output_tokens and other existing throughput stats in this same file are handled (no zero-filtering there either), so this is consistent with existing conventions rather than a new defect.
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
cquil11
changed the title
cquil11
added
agentx-fast
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
d368c20 to
1985ac7
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31045023380 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31045242272 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31052700355 |
1 participant
Summary
agentx-v1.0.2at the merged full-response implementationfull_response_inter_token_latencyfrom AgentX per-request profile recordsrequest_metrics.latency.full_response_itlrequest_metrics.latency.full_response_intvtyWhy
AgentX can receive a server-reported raw token count even when a structured-output parser stops emitting visible content before generation completes. The existing ITL then mixes the raw token count with a shorter parsed-content interval.
The full-response ITL uses the client-observed interval through HTTP response completion. InferenceX derives each interactivity statistic as the reciprocal of its matching ITL statistic, consistent with the existing
itlandintvtyfields. For example, full-response p75 interactivity is1 / p75(full-response ITL), preserving slow-tail percentile semantics.Validation
PYTHONPATH=. uv run pytest -q utils/agentic/aggregation: 43 passedgit diff --check: passedDependencies and merge order
agentx-v1.0.2)