Skip to content

(1) Implement generic oak_cache#1298

Open
DavisVaughan wants to merge 1 commit into
mainfrom
feature/sources-1-oak-cache
Open

(1) Implement generic oak_cache#1298
DavisVaughan wants to merge 1 commit into
mainfrom
feature/sources-1-oak-cache

Conversation

@DavisVaughan

Copy link
Copy Markdown
Contributor

Part of #1234

Part of a series of PRs to get the sources mechanism integrated with oak. Most of the PRs revolve around restructuring the cache in a way that I think makes a lot more sense (particularly if we want it to be usable elsewhere).

We end up with:

  • oak_cache - Generic Cache mechanism
  • oak_srcref - Home to SrcrefCache, requires R, extracts the srcref attribute
  • oak_source - Home to SourceCache, which knows how to download CRAN R package tarballs and R version tarballs

And oak_sources gets removed at the end.

When I worked on the caching stuff earlier, I struggled to reconcile with the fact that different caching sources return different things. We have 3 things we can return:

  • srcrefs, which is just the reconstructed R/ folder
  • CRAN R package tarballs, which is a single R package in its entirety
  • CRAN R version tarballs, which is a single R version in its entirety

Rather than trying to squish these together into a single cache mechanism, which required us to drop useful things from the CRAN downloads to align with the more minimal result we get from the srcref extractor, we now keep them separate in separate caches, so we end up with:

  • oak_srcref::SrcrefCache feeding to .cache/oak/srcref/
  • oak_source::SourceCache::cran feeding to .cache/oak/source/cran/
  • oak_source::SourceCache::r feeding to .cache/oak/source/r/

Each cache folder now has a consistent structure that also maximizes information retention. i.e. in the CRAN download we keep the entire R package source, including things like man/, src/, and any inst/ folder you might have included. I imagine this could be useful in the future (navigating to C functions?).

I'm also hoping that oak_source can be useful on its own without requiring the heavier R dependency that oak_srcref has.

The end result of this series of 5 PRs is a production level SourceHandler that is integrated in the LSP that pulls sources using oak_srcref and oak_source when you are in a debug build. Release builds still don't have it turned on yet. And it isn't being utilized in any LSP features yet. I'll turn them on one by one after merging this series.

Comment on lines +24 to +28
/// Its mtime doubles as the entry's last-access time. [`Cache::get`] bumps it on every
/// retrieval (this is an atomic operation so we allow it even though we'll only hold a
/// shared lock on this key's folder) and [`Cache::clean`] evicts the entries with the
/// oldest mtime first.
const COMPLETE_FILENAME: &str = ".complete";

@DavisVaughan DavisVaughan Jun 29, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps the most interesting change is that we've switched the "cleanup" mechanism to a simpler LRU idea, kind of like salsa.

(Previously it would try and track the original libpath the request came from, and would cleanup if the package no longer existed in that libpath. It was too complicated. Plus, oak_source doesn't actually even take a libpath as an input anymore!)

We keep a max of N entries in the cache, where the mtime of the .complete completion sentinel file is our way of detecting "last use time" to order them.

We "touch" .complete on every get() of that package to keep it updated. Even though we only hold a shared lock on the folder, this one modification is fine because it is atomic, and there is a guarantee that no one can delete the file out from under us.

We keep at most:

  • 200 srcrefs
  • 200 R package sources
  • 5 R versions

@DavisVaughan DavisVaughan requested a review from lionel- June 29, 2026 13:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant