Adopt content-addressed SQLite cache with HF sync - #77
Conversation
There was a problem hiding this comment.
Thanks for doing this @ErlisLushtaku ! Just had a first read, did not finish it, I will finish the PR in a second pass but already have some comments.
One thing that occurs to me as well, the storage seems to be a bit leaking in the sense that all mains needs to be modified and it seems that we must make them aware of the caching.
However, ideally I think we could have something which is more like:
completion_fun = make_model(model, cache_args)
annotation_fun = make_model(judge_model, cache_args)
and then call those in any entrypoint with something like
completions, metadata = completion_fun(
inputs=...,
hash=...,
)
which would feel less invasive and hopefully will reduce the LOC change.
|
|
||
| ### Shared SQLite cache | ||
|
|
||
| Pass `--store_root <path>` to reuse completion and judgement rows across runs. |
There was a problem hiding this comment.
I think adding a high level description before would make sense, something like:
| Pass `--store_root <path>` to reuse completion and judgement rows across runs. | |
| Storing completions and judgement annotations can be made so that LLM calls are avoided by reading from disk. Completions and judgements are written into `~/judgearena-data/db/completions/{task}/{model_name}/{provider}/{config_hash}/` and can be deleted by the user (judgements can be accessed in judgements/). In addition, completions can be fetched from a shared Hugginface (fetching is activated by default, sharing is not). | |
| Pass `--store_root <path>` to reuse completion and judgement rows across runs. |
There was a problem hiding this comment.
we may not need that one right?
| @@ -1,4 +1,4 @@ | |||
| """Benchmark SQLiteCompletionStore / SQLiteJudgementStore load and missing_indices speed. | |||
| """Benchmark SQLite completion/judgement store lookup speed. | |||
There was a problem hiding this comment.
Perhaps we move those to scripts/db/ to avoid spamming scripts/?
| max_model_len=args.max_model_len, | ||
| chat_template=args.chat_template, | ||
| result_folder=args.result_folder, | ||
| store_root=args.store_root, |
There was a problem hiding this comment.
would it be clear to add those to a namespace with a cache config?
eg to add args.cache_config with its own dataclass which contains those 6 fields.
| return Path(store_root) / kind / task / model_name / provider | ||
| def _cache_stores( | ||
| args: CliEloArgs, | ||
| ) -> tuple[ |
There was a problem hiding this comment.
We almost never wants to return a tuple with 5 elements.
Ideally, we splits into subfunctions which do different things, if not possible we should do a dataclasses (but here it really looks like we could do separate functions, for instance, it would be cleaner to pass judge_system_prompt, judge_user_prompt as input and we have already 2 outputs less.
| if args.store_root is not None: | ||
| comp_folder = _store_folder( | ||
| args.store_root, "completions", args.arena, args.model | ||
| def _collapse_elo_prefs(prefs: list, n_battles: int) -> list: |
There was a problem hiding this comment.
| def _collapse_elo_prefs(prefs: list, n_battles: int) -> list: | |
| def _collapse_elo_prefs(prefs[float]: list, n_battles: int) -> list[float]: |
?
| comp_folder = _store_folder( | ||
| args.store_root, "completions", args.arena, args.model | ||
| def _collapse_elo_prefs(prefs: list, n_battles: int) -> list: | ||
| if len(prefs) == n_battles: |
There was a problem hiding this comment.
The contract of this function is a bit weird, it feels to me that this would be cleaner:
# in the caller
if args.swap_mode:
# average the preferences which are the concatentation of preference with A and B as model and then reverse
prefs = _collapse_pairwise_prefs(raw_prefs)
else:
prefs = raw_prefs | return collapsed | ||
|
|
||
|
|
||
| def _cache_elo_direction( |
There was a problem hiding this comment.
The name is not super telling, perhaps _cache_elo_judgement?
Introduce content-addressed SQLite cells with atomic metadata associations and deterministic Hugging Face merge semantics. Includes-AI-Code: true
Add lazy prepared-model adapters and route inference through versioned, provider-aware descriptors without initializing backends on full hits. Includes-AI-Code: true
Thread one inference cache through generation, judging, ELO, MT-Bench, and meta-eval while removing the legacy pass-level cache paths. Includes-AI-Code: true
Expose strict cache synchronization commands and conservatively reconstruct hosted judge rows from verifiable saved-run artifacts. Includes-AI-Code: true
Describe cache modes, hosted-provider limits, offline Hugging Face synchronization, SQLite constraints, and conservative migration examples. Includes-AI-Code: true
cce746f to
75c7699
Compare
2 participants
Summary
judgearena-cache).