Ensure nested allocations in statics neither get deduplicated nor duplicated #121644

oli-obk · 2024-02-26T18:09:25Z

This PR generates new DefIds for nested allocations in static items and feeds all the right queries to make the compiler believe these are regular static items. I chose this design, because all other designs are fragile and make the compiler horribly complex for such a niche use case.

At present this wrecks incremental compilation performance in case nested allocations exist (because any query creating a DefId will be recomputed and never loaded from the cache). This will be resolved later in #115613 . All other statics are unaffected by this change and will not have performance regressions (heh, famous last words)

This PR contains various smaller refactorings that can be pulled out into separate PRs. It is best reviewed commit-by-commit. The last commit is where the actual magic happens.

r? @RalfJung on the const interner and engine changes

fixes #79738

oli-obk · 2024-02-26T18:09:34Z

@bors try @rust-timer queue

rustbot · 2024-02-26T18:09:40Z

Some changes occurred to the CTFE / Miri engine

cc @rust-lang/miri

Some changes occurred in match checking

cc @Nadrieril

Some changes occurred in compiler/rustc_codegen_gcc

cc @antoyo, @GuillaumeGomez

Some changes occurred to MIR optimizations

cc @rust-lang/wg-mir-opt

Some changes occurred in src/tools/clippy

cc @rust-lang/clippy

Some changes occurred to the CTFE / Miri engine

cc @rust-lang/miri

bors · 2024-02-26T18:10:45Z

⌛ Trying commit 0aa7752 with merge 51f4acc...

Ensure nested allocations in statics do not get deduplicated This PR generates new `DefId`s for nested allocations in static items and feeds all the right queries to make the compiler believe these are regular `static` items. I chose this design, because all other designs are fragile and make the compiler horribly complex for such a niche use case. At present this wrecks incremental compilation performance *in case nested allocations exist* (because any query creating a `DefId` will be recomputed and never loaded from the cache). This will be resolved later in rust-lang#115613 . All other statics are unaffected by this change and will not have performance implications (heh, famous last words) This PR contains various smaller refactorings that can be pulled out into separate PRs. It is best reviewed commit-by-commit. The last commit is where the actual magic happens. r? `@RalfJung` on the const interner and engine changes

bors · 2024-02-26T18:26:18Z

💔 Test failed - checks-actions

oli-obk · 2024-02-26T20:24:07Z

@bors try @rust-timer queue

bors · 2024-02-26T20:40:32Z

⌛ Trying commit cc1ec8a with merge 2e8adb0...

Ensure nested allocations in statics do not get deduplicated This PR generates new `DefId`s for nested allocations in static items and feeds all the right queries to make the compiler believe these are regular `static` items. I chose this design, because all other designs are fragile and make the compiler horribly complex for such a niche use case. At present this wrecks incremental compilation performance *in case nested allocations exist* (because any query creating a `DefId` will be recomputed and never loaded from the cache). This will be resolved later in rust-lang#115613 . All other statics are unaffected by this change and will not have performance implications (heh, famous last words) This PR contains various smaller refactorings that can be pulled out into separate PRs. It is best reviewed commit-by-commit. The last commit is where the actual magic happens. r? `@RalfJung` on the const interner and engine changes fixes rust-lang#79738

bors · 2024-02-26T21:05:18Z

💔 Test failed - checks-actions

oli-obk · 2024-02-26T23:33:48Z

@bors try @rust-timer queue

bors · 2024-02-26T23:34:59Z

⌛ Trying commit e5836ab with merge ee17466...

Ensure nested allocations in statics do not get deduplicated This PR generates new `DefId`s for nested allocations in static items and feeds all the right queries to make the compiler believe these are regular `static` items. I chose this design, because all other designs are fragile and make the compiler horribly complex for such a niche use case. At present this wrecks incremental compilation performance *in case nested allocations exist* (because any query creating a `DefId` will be recomputed and never loaded from the cache). This will be resolved later in rust-lang#115613 . All other statics are unaffected by this change and will not have performance regressions (heh, famous last words) This PR contains various smaller refactorings that can be pulled out into separate PRs. It is best reviewed commit-by-commit. The last commit is where the actual magic happens. r? `@RalfJung` on the const interner and engine changes fixes rust-lang#79738

oli-obk · 2024-03-12T08:51:40Z

@bors r=RalfJung,nnethercote

bors · 2024-03-12T08:51:43Z

📌 Commit e277373 has been approved by RalfJung,nnethercote

It is now in the queue for this repository.

bors · 2024-03-12T10:29:19Z

⌛ Testing commit e277373 with merge 3b85d2c...

bors · 2024-03-12T12:52:24Z

☀️ Test successful - checks-actions
Approved by: RalfJung,nnethercote
Pushing 3b85d2c to master...

rust-timer · 2024-03-12T14:38:17Z

Finished benchmarking commit (3b85d2c): comparison URL.

Overall result: ✅ improvements - no action needed

@rustbot label: -perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-0.2%	[-0.2%, -0.2%]	1
All ❌✅ (primary)	-	-	0

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	9.6%	[9.6%, 9.6%]	1
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-5.1%	[-5.1%, -5.1%]	1
All ❌✅ (primary)	9.6%	[9.6%, 9.6%]	1

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	3.9%	[3.8%, 4.1%]	2
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-3.0%	[-3.0%, -3.0%]	1
All ❌✅ (primary)	-	-	0

Binary size

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.0%	[0.0%, 0.0%]	4
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.0%	[0.0%, 0.0%]	4

Bootstrap: 672.558s -> 674.911s (0.35%)
Artifact size: 309.99 MiB -> 310.31 MiB (0.10%)

oli-obk · 2024-03-13T09:18:43Z

compiler/rustc_const_eval/src/const_eval/eval_queries.rs

+
+            // Temporarily allow access to the static_root_ids for the purpose of validation.
+            let static_root_ids = ecx.machine.static_root_ids.take();
            let res = const_validate_mplace(&ecx, &mplace, cid);
+            ecx.machine.static_root_ids = static_root_ids;


in case anyone stumbles upon this in the future, this was a rebase screwup and is getting removed again in #122397 (comment)

bjorn3 · 2024-03-18T16:06:26Z

tests/ui/consts/const-mut-refs-crate.rs

+pub fn main() {
+    unsafe {
+        assert_eq!(FOO as *const i32, BAR as *const i32);
+        assert_eq!(INNER_MOD_FOO as *const i32, INNER_MOD_BAR as *const i32);


Looks like this case still depends on deduplication of anonymous constants, causing the test to fail with cg_clif.

yea it's more like a canary test than anything we want to necessarily guarantee. Though maybe we could start guaranteeing the deduplication by generating canonical static items whose id/name is their contents' stable hash?

yea it's more like a canary test than anything we want to necessarily guarantee.

Would it be possible to split the test into a guaranteed half and a canary half? That way the guaranteed half can be tested with cg_clif and the canary half can be ignored.

yea that sounds reasonable. Will open a PR shortly

wait hold up, this is all guaranteed. I mistakenly thought there was deduplication due to some constants happening, but it's all static items. there is no deduplication. Instead these are defined as

pub mod inner { pub static INNER_MOD_FOO: &'static i32 = &43; pub static INNER_MOD_BAR: &'static i32 = INNER_MOD_FOO; }

where we would expect them to be the same

At the MIR level, INNER_MOD_FOO has 43 in promoted MIR, while INNER_MOD_BAR has it in an already evaluated anonymous allocation:

static INNER_MOD_FOO: &i32 = { let mut _0: &i32; let _1: &i32; let _2: i32; let mut _3: &i32; bb0: { StorageLive(_1); _3 = const INNER_MOD_FOO::promoted[0]; _1 = &(*_3); _0 = &(*_1); StorageDead(_1); return; } } const INNER_MOD_FOO::promoted[0]: &i32 = { let mut _0: &i32; let mut _1: i32; bb0: { _1 = const 43_i32; _0 = &_1; return; } } static INNER_MOD_BAR: &i32 = { let mut _0: &i32; let mut _1: &&i32; bb0: { StorageLive(_1); _1 = const {alloc1: &&i32}; _0 = (*_1); StorageDead(_1); return; } } alloc1 (static: INNER_MOD_FOO, size: 8, align: 8) { ╾─────alloc3<imm>─────╼ │ ╾──────╼ } alloc3 (size: 4, align: 4) { 2b 00 00 00 │ +... }

oh right, this is using promotion 🤦 Does the test work for you if you change the body of INNER_MOD_FOO to

{ let x = 43; &{x} }

That will test that specific assertion. For the rest equivalent changes are necessary.

Some tests maybe also should be swapped to mutable references. The test is called "const-mut-refs" after all. ;)

…n-attrs, r=oli-obk Don't inherit codegen attrs from parent static Putting this up partly for discussion and partly for review. Specifically, in rust-lang#121644, `@oli-obk` designed a system that creates new static items for representing nested allocations in statics. However, in that PR, oli made it so that these statics inherited the codegen attrs from the parent. This causes problems such as colliding symbols with `#[export_name]` and ICEs with `#[no_mangle]` since these synthetic statics have no `tcx.item_name(..)`. So the question is, is there any case where we *do* want to inherit codegen attrs from the parent? The only one that seems a bit suspicious is the thread-local attribute. And there may be some interesting interactions with the coverage attributes as well... Fixes (after backport) rust-lang#123274. Fixes rust-lang#123243. cc rust-lang#121644. r? `@oli-obk` cc `@nnethercote` `@RalfJung` (reviewers on that pr)