Sparse lock files WIP - #579
Open
edolstra wants to merge 10 commits into
Open
Conversation
In preparation for the version 8 sparse lock file format, rename LockFile to LockFileV7 and move it, together with the Node and LockedNode data types, to a new header lockfile-v7.hh (implementation in lockfile-v7.cc). lockfile.hh continues to contain the common types like InputAttrPath and NonEmptyInputAttrPath. No behaviour change. Assisted-by: Claude Fable 5 <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
In preparation for the version 8 sparse lock file format, turn `LockedFlake` into an abstract base class that doesn't expose the version 7 lock file representation, and move that representation into a new subclass `LockedFlakeV7`. The `lockFlake()` functions now return a `std::unique_ptr<LockedFlake>`. Code that needs to query a locked flake now goes through virtual methods instead of inspecting the version 7 node graph directly: * `isUnlocked()`: check for unlocked or non-final inputs. This allows `getFingerprint()` to be implemented entirely in terms of the abstract interface. * `toJSON()`, with non-virtual `to_string()` and `operator <<` on top of it. * `getInputNames()`: return the names of the inputs of the input denoted by an attribute path prefix. * `findInput()`: look up an input by attribute path, resolving "follows" indirections, and return an `InputInfo` struct containing its locked reference. Used by `--inputs-from` and `InstallableFlake::nixpkgsFlakeRef()`. * `visit()`: walk all transitive inputs in depth-first order, calling a callback with either an `InputInfo` or the target path of a "follows" input. The callback controls recursion, e.g. to skip build-time inputs. Used by `nix flake metadata`, `nix flake archive` and `nix flake prefetch-inputs`. The only remaining uses of `LockedFlakeV7` are in `lockFlake()` and `callFlake()` in libflake itself, which will become the dispatch points between the version 7 and version 8 implementations. Assisted-by: Claude Fable 5 <noreply@anthropic.com>
The `computeLocks` machinery is moved from the free function `lockFlake()` in flake.cc into a new static member function `LockedFlakeV7::lockFlake()` in lockfile-v7.cc. The free function now only contains version-independent work, expressed in terms of the abstract `LockedFlake` interface: * Reading the old lock file into a `nlohmann::json`, and constructing a `LockedFlake` from it (via a new `LockedFlakeV7` constructor that takes the JSON). This is where we will dispatch on the lock file version in the future. * Change detection, by comparing the virtual `toJSON()` serializations of the old and new locked flakes. This also does the right thing across future version migrations (a version change is always a change). * Printing the diff, via a new abstract `LockedFlake::diff()` method that shows the differences relative to an older `LockedFlake`. The version 7 implementation diffs against an empty lock file if the old lock file is not version 7, so all inputs show up as added. * Writing/committing the new lock file, using `to_string()` and the virtual `isUnlocked()`. `LockedFlakeV7::lockFlake()` takes the old lock file as a `const LockedFlake &` and downcasts it internally. The `LockFileV7` parsing constructor now takes a `nlohmann::json` instead of the file contents, and `getFlake()` with a lock root attribute path is exported from flake.cc since the lock algorithm needs it from its new home. Assisted-by: Claude Fable 5 <noreply@anthropic.com>
This abstract method returns the source path of the input denoted by an input attribute path (or of the top-level flake if the path is empty), fetching the input if necessary. The returned path is backed by `EvalState::rootFS`, i.e. it's a store path - possibly a virtual one with the input's accessor mounted on it if lazy trees are enabled. `nix flake prefetch-inputs` now uses this method instead of accessing the version 7 node graph directly, so when lazy trees are disabled, inputs are copied to the store via `EvalState::mountInput()` (which also verifies their NAR hashes). In `LockedFlakeV7`, the `nodePaths` map is replaced by a `mutable Sync<std::optional<SourcePath>>` field in `LockedNode`, i.e. the source path of a fetched node is now stored in the node itself. It is set by `LockedFlakeV7::lockFlake()` for nodes fetched during locking, and cached by `getSourcePath()` for nodes fetched on demand (where relative path inputs are resolved against the source path of their parent flake, and the `subdir` of the locked flakeref is taken into account). `callFlake()` reconstructs the node -> source path map by walking the node graph. Assisted-by: Claude Fable 5 <noreply@anthropic.com>
edolstra
added
the
flake-regression-test
Instead of the version 7 lock file JSON and an attrset of pre-fetched
source trees ("overrides"), call-flake.nix now receives an external
value wrapping the C++ `LockedFlake` object, along with two internal
primops (registered with `internal = true`, so they're not exposed to
the user):
* `listFlakeInputs lockedFlake inputAttrPath`: returns the inputs of
the input denoted by `inputAttrPath` as an attrset mapping input
names to either null (for a regular input) or the input attribute
path of the target of a "follows" input. This only consults the
lock data, so it doesn't fetch anything.
* `fetchFlakeInput lockedFlake inputAttrPath`: fetches an input (via
`LockedFlake::getSourcePath()`) and returns its `sourceInfo`
attributes and flake subdirectory; or, for build-time inputs, the
locked input attributes (without fetching).
As a result, fetching, override handling, relative path handling and
the lock file node graph all disappear from call-flake.nix - it just
lazily constructs a tree of inputs keyed by input attribute path,
resolving "follows" by walking the edges of that tree from the
top-level flake. This preserves evaluation sharing (every distinct
input is constructed only once) and prepares for the version 8 lock
file format, whose `LockedFlake` implementation will only need to
provide the same abstract interface.
Also:
* `callFlake()` now takes a `std::shared_ptr<const LockedFlake>`
since the thunks created by call-flake.nix reference the
`LockedFlake` object, which therefore must be kept alive for the
lifetime of the evaluator.
* `LockedFlake::getInputTargets()` is a new abstract method backing
`listFlakeInputs`; `getInputNames()` is now a non-virtual wrapper
around it.
* `LockedFlakeV7::findInput()` now returns the `buildTime` and
`parentInputAttrPath` fields (the former was dropped before, but
wasn't used).
* The `KeyMap` returned by `LockFileV7::toJSON()` is no longer
needed, since lock file node keys were only used by callFlake().
Assisted-by: Claude Fable 5 <noreply@anthropic.com>
These types are no longer used outside of lockfile-v7.cc, so they can
be implementation details. The header now only exposes two free
functions returning `std::unique_ptr<LockedFlake>`:
* `parseLockFileV7()`: construct a `LockedFlake` from the JSON
contents of a version 5-7 lock file.
* `lockFlakeV7()` (previously `LockedFlakeV7::lockFlake()`): compute a
new lock file.
This also keeps the version dispatch in the free function `lockFlake()`
symmetrical for the future version 8 implementation: it will just pick
between `parseLockFileV{7,8}()` and `lockFlakeV{7,8}()` based on the
`version` field in the lock file JSON.
Also remove the unused `LockedNode::computeStorePath()`.
Assisted-by: Claude Fable 5 <noreply@anthropic.com>
Now that these classes are local to this file, there is no need for separate declarations and definitions. Assisted-by: Claude Fable 5 <noreply@anthropic.com>
It's the input attribute path, relative to the top-level flake, of the flake that *declares* the relative path (i.e. the flake that declares the override, in the case of overridden inputs), which is not necessarily the input's parent in the lock file graph. Assisted-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Context