fix: estimator cache memory, RestorationTracker bounds, remove dead ParallelExecutor - #81
Merged
Conversation
The token cache stored the entire original string in every cacheEntry for collision checks, and hashText copied each input to []byte on every Count. Since Compress estimates the original prompt plus every intermediate layer output, large sessions could retain hundreds of MB in the 8192-entry LRU. - Hash with hash/maphash.String seeded once per process: zero-copy, and a random seed removes fixed-seed hash-flooding concerns. - Validate hits by (hash, length); drop the text field from cacheEntry. Collision odds (~1e-12 per process) are documented in hashText. - Skip the cache entirely for texts above maxCacheableTextBytes (1 MiB) so giant inputs never enter the LRU. - Concurrency shape (64 shards) unchanged. Tests: hit across calls, same-length/different-text miss, oversize bypass at and around the cutoff, and a reflection guard that cacheEntry carries no string field.
Track() did a linear scan over all entries per call (O(n^2) across a session) and retained every tracked file's full Content until Clear(). A long session tracking large files grew the tracker without bound. - Dedup via a path-indexed map backed by a doubly-linked list: O(1) Track, and re-tracking a path refreshes its recency. - Bound total retained content by a token budget (DefaultRetainedTokenBudget = 200_000 tokens, one default context window ~ 800 KB); the least recently tracked entries are evicted when exceeded. Configurable through the existing builder pattern via WithRetainedTokenBudget, which also evicts immediately when lowered. The most recently tracked entry is always kept. - Clear() resets the list, index, and token total. Tests: repeated same-path tracks collapse to one latest-wins entry, budget eviction drops the oldest entry, refreshed entries survive over stale ones, a single oversize entry is kept, lowering the budget evicts retroactively, and Clear semantics hold.
ParallelExecutor ran every layer on the full input concurrently and kept the shortest output — bytes, not tokens. It has no production callers in tok or hawk and no test coverage; the token-budget pipeline cannot use it as-is. Removing to keep the filter surface honest.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
maphash.Stringhashing (hash+length for collision check); added 1 MiB cache-size cutoff. Eliminates multi-hundred-MB retention on large prompts.Test plan
GOWORK=off go build ./...+go vet ./...GOWORK=off go test ./...— 923 tests, 0 fail, 16 packages