perf: avoid repeated upstream session context summarization

[ChanX21]

Area

Judge / Scoring

Problem or motivation

Memory-dependent evaluator flows can reuse the same upstream session context multiple times during a run. When that upstream transcript exceeds the inline character budget, formatUpstreamSessions() summarizes it with an LLM.

Currently, repeated calls with identical upstream context can trigger repeated summarizer LLM calls, adding avoidable latency and cost.

Proposed solution

Cache successful formatted upstream session summaries per model instance using a bounded in-memory cache.

The cache should preserve correctness by keying on:

• rendered upstream context
• max character budget
• label style
• section header

Fallback output from failed summarization should not be cached.

And expected impact from this

• Reduces duplicate LLM summarization calls in memory plant-to-trigger and other dependent evaluator flows.
• Keeps prompt behavior unchanged.
• Adds a regression test proving repeated identical large upstream context is summarized once.

Alternatives considered

considered hashing the cache key, but considering the tradeoff of hash collision better keep the full rendered context for correctness over a small memory optimization for this bounded cache.

Would you like to contribute this?

Yes — I'd like to submit a PR