(1) Implement generic oak_cache#1298
Conversation
| /// Its mtime doubles as the entry's last-access time. [`Cache::get`] bumps it on every | ||
| /// retrieval (this is an atomic operation so we allow it even though we'll only hold a | ||
| /// shared lock on this key's folder) and [`Cache::clean`] evicts the entries with the | ||
| /// oldest mtime first. | ||
| const COMPLETE_FILENAME: &str = ".complete"; |
There was a problem hiding this comment.
Perhaps the most interesting change is that we've switched the "cleanup" mechanism to a simpler LRU idea, kind of like salsa.
(Previously it would try and track the original libpath the request came from, and would cleanup if the package no longer existed in that libpath. It was too complicated. Plus, oak_source doesn't actually even take a libpath as an input anymore!)
We keep a max of N entries in the cache, where the mtime of the .complete completion sentinel file is our way of detecting "last use time" to order them.
We "touch" .complete on every get() of that package to keep it updated. Even though we only hold a shared lock on the folder, this one modification is fine because it is atomic, and there is a guarantee that no one can delete the file out from under us.
We keep at most:
- 200 srcrefs
- 200 R package sources
- 5 R versions
Part of #1234
Part of a series of PRs to get the sources mechanism integrated with oak. Most of the PRs revolve around restructuring the cache in a way that I think makes a lot more sense (particularly if we want it to be usable elsewhere).
We end up with:
oak_cache- GenericCachemechanismoak_srcref- Home toSrcrefCache, requires R, extracts thesrcrefattributeoak_source- Home toSourceCache, which knows how to download CRAN R package tarballs and R version tarballsAnd
oak_sourcesgets removed at the end.When I worked on the caching stuff earlier, I struggled to reconcile with the fact that different caching sources return different things. We have 3 things we can return:
R/folderRather than trying to squish these together into a single cache mechanism, which required us to drop useful things from the CRAN downloads to align with the more minimal result we get from the srcref extractor, we now keep them separate in separate caches, so we end up with:
oak_srcref::SrcrefCachefeeding to.cache/oak/srcref/oak_source::SourceCache::cranfeeding to.cache/oak/source/cran/oak_source::SourceCache::rfeeding to.cache/oak/source/r/Each cache folder now has a consistent structure that also maximizes information retention. i.e. in the CRAN download we keep the entire R package source, including things like
man/,src/, and anyinst/folder you might have included. I imagine this could be useful in the future (navigating to C functions?).I'm also hoping that
oak_sourcecan be useful on its own without requiring the heavier R dependency thatoak_srcrefhas.The end result of this series of 5 PRs is a production level
SourceHandlerthat is integrated in the LSP that pulls sources usingoak_srcrefandoak_sourcewhen you are in a debug build. Release builds still don't have it turned on yet. And it isn't being utilized in any LSP features yet. I'll turn them on one by one after merging this series.