perf: Keep dep-graph indices stable and re-emit unchanged nodes

[xmakro]

View all comments

On a warm rebuild most of the previous graph is still valid. Every one of those nodes is decoded and then encoded again, into a record that differs from the one on disk only because the indices were handed out anew.

In this PR, we keep those nodes at their previous index. A node that is still valid only points at other nodes that are still valid, and those keep their indices too. So its edges do not change, and its whole record is byte for byte the one already on disk. Promoting a node now means writing its old record again instead of building it a second time. Decoding does not keep the header, so we pack it again from the fields we decoded, but the edges are copied across still encoded. Most of what this skips is per edge: scanning the list to pick an index width and writing each index out in that width.

When a node disappears, its index is left unoccupied, and the next session hands unoccupied indices to new nodes before extending the index space. That cannot disturb a promoted record: its edges point only at nodes that were live when the file was written, so nothing in the file references an unoccupied index. Indices a thread takes in a batch and does not use come back the same way, so batch allocation stays as it is.

Stable indices also shrink the color map. A green node's current index equals its previous index, so the map no longer stores it: one byte per node, green, red or unknown, a quarter of its former size.

On disk this costs almost nothing: records exist only for live nodes, so an unoccupied index has no bytes in the file. Measured on a synthetic crate over ten generations of add/remove churn, the dep-graph file is about 1% larger than before and stays flat as nodes come and go. The cost is in memory at load time: the decode tables are sized by the index space, at roughly 50 bytes per unoccupied index, and the index space tracks the most nodes the graph has ever held at once.

Much of the motivation of this PR is what stable indices allow as next perf improvements:

• Green marking gathers an edge list for every node it promotes. Only the debugging graph behind -Zquery-dep-graph reads that list, so the walk can stop building it otherwise.
• The previous file can be mapped into memory and a promoted record's bytes copied straight across, instead of packing its header and copying its edges out of the decoded arrays.
• With that memory mapping retained for the whole session, edge headers can hold file positions and edge lists can be served from the mapped file in place, so decoding stops copying edge bytes as well.
• The save can go further and keep the previous record region as one block, written out in a single call, with new and re-executed records appended after it and a later record at an index replacing the earlier one. A save then writes in proportion to what changed, not to the size of the graph, at the cost of replaced and deleted records staying in the file until some session rewrites it.
• The query result cache stores each green node's value under its dep node index. Once indices hold still across sessions, those values can be carried the same way, instead of every cached value being decoded and encoded again on each save.

[Kobzol]

@bors try @rust-timer queue

[rust-bors]

☀️ Try build successful (CI)
Build commit: 5c4fe74 (5c4fe748da88211f53774d6c125545b6112b74f2)
Base parent: be3d26d (be3d26db984c6f96335faca1f254dc04873cb1c1)

[rust-timer]

Finished benchmarking commit (5c4fe74): comparison URL.

Overall result: ✅ improvements - no action needed

Benchmarking means the PR may be perf-sensitive. It's automatically marked not fit for rolling up. Overriding is possible but disadvised: it risks changing compiler perf.

@bors rollup=never rustc-perf
@rustbot label: -S-waiting-on-perf -perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	0.1%	[0.1%, 0.1%]	1
Regressions ❌ (secondary)	0.2%	[0.1%, 0.3%]	8
Improvements ✅ (primary)	-1.7%	[-4.7%, -0.2%]	109
Improvements ✅ (secondary)	-1.4%	[-3.7%, -0.2%]	60
All ❌✅ (primary)	-1.7%	[-4.7%, 0.1%]	110

Max RSS (memory usage)

Results (primary 2.1%, secondary -0.7%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	2.1%	[2.1%, 2.1%]	1
Regressions ❌ (secondary)	0.4%	[0.4%, 0.4%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-1.8%	[-1.8%, -1.8%]	1
All ❌✅ (primary)	2.1%	[2.1%, 2.1%]	1

Cycles

Results (primary -0.1%, secondary -1.8%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	0.8%	[0.5%, 1.4%]	7
Regressions ❌ (secondary)	1.2%	[0.4%, 3.0%]	12
Improvements ✅ (primary)	-2.1%	[-2.8%, -1.3%]	3
Improvements ✅ (secondary)	-7.8%	[-13.7%, -2.4%]	6
All ❌✅ (primary)	-0.1%	[-2.8%, 1.4%]	10

Binary size

This perf run didn't have relevant results for this metric.

Bootstrap: 489.155s -> 493.076s (0.80%)
Artifact size: 390.64 MiB -> 390.60 MiB (-0.01%)

[xmakro]

r? @cjgillot

[cjgillot]

Thanks a lot for the PR. It increases substantially the cognitive complexity of node encoding, and I already have trouble wrapping my head over this.

There was an old design decision around incremental to avoid depending on history: the contents of the dep-graph or on-disk cache should not be modified by the previous dep-graph or cache. This was meant to ease debugging, in particular if a user comes with an ICE after a dozen incremental recompiles.

This PR breaks that design. This may need a MCP to gather thoughts on this.

View changes since this review

[cjgillot]

When can this happen?

[xmakro]

Added comment. I think it is the same situation the existing comment in try_mark_previous_green describes:

rust/compiler/rustc_middle/src/dep_graph/graph.rs

Line 1018 in 58b7197

// `no_hash` nodes may fail this promotion due to already being conservatively colored red.

[rustbot]

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

[xmakro]

Thank you for the great review, as always.

The new stable index invariant also allowed another cleanup: DepNodeColorMap no longer needs a green node's index, since it equals the previous index. The map is now one byte per node (green, red, unknown) instead of 4-bytes, and DesiredColor::Green and TrySetColorResult::AlreadyGreen lost their index payloads. This should give marking 4x more nodes per cache line, would be interesting to see another rustc-perf run.

On the history dependence: is the concern the stable indices, or bytes flowing from the old file into the new one? In this PR everything that gets written has passed through decode. Only the edges come from edge_list_data instead of being re-encoded. But the rule would be broken in the follow-ups, where we would copy raw records straight across.

[jhpratt]

@bors try @rust-timer queue

[rust-bors]

☀️ Try build successful (CI)
Build commit: e66ed41 (e66ed41f1c397cf2a87164d2e78de5edc39d5023)
Base parent: 1ed2df6 (1ed2df61a19042f231709eb05d032ae9e2cb2084)

[rust-timer]

Finished benchmarking commit (e66ed41): comparison URL.

Overall result: ❌✅ regressions and improvements - please read:

Benchmarking means the PR may be perf-sensitive. It's automatically marked not fit for rolling up. Overriding is possible but disadvised: it risks changing compiler perf.

Next, please: If you can, justify the regressions found in this try perf run in writing along with @rustbot label: +perf-regression-triaged. If not, fix the regressions and do another perf run. Neutral or positive results will clear the label automatically.

@bors rollup=never rustc-perf
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	0.3%	[0.3%, 0.4%]	5
Regressions ❌ (secondary)	0.2%	[0.2%, 0.2%]	2
Improvements ✅ (primary)	-2.1%	[-6.3%, -0.2%]	115
Improvements ✅ (secondary)	-1.8%	[-5.3%, -0.4%]	64
All ❌✅ (primary)	-2.0%	[-6.3%, 0.4%]	120

Max RSS (memory usage)

Results (primary -1.4%, secondary -0.1%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	5.8%	[4.3%, 8.0%]	3
Improvements ✅ (primary)	-1.4%	[-5.8%, -0.6%]	12
Improvements ✅ (secondary)	-3.1%	[-4.6%, -0.4%]	6
All ❌✅ (primary)	-1.4%	[-5.8%, -0.6%]	12

Cycles

Results (primary -2.0%, secondary 0.5%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	1.2%	[0.5%, 2.2%]	3
Regressions ❌ (secondary)	7.4%	[0.6%, 34.7%]	6
Improvements ✅ (primary)	-2.4%	[-5.0%, -0.4%]	28
Improvements ✅ (secondary)	-1.9%	[-3.7%, -0.5%]	17
All ❌✅ (primary)	-2.0%	[-5.0%, 2.2%]	31

Binary size

Results (secondary -0.0%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-0.0%	[-0.0%, -0.0%]	1
All ❌✅ (primary)	-	-	0

Bootstrap: 489.825s -> 491.849s (0.41%)
Artifact size: 391.14 MiB -> 391.34 MiB (0.05%)