Add chapter on fuzzing #1646

langston-barrett · 2023-03-16T02:34:41Z

src/fuzzing.md

compiler-errors · 2023-03-16T04:24:39Z

src/fuzzing.md

+If you're not sure whether or not an ICE is a duplicate of one that's already
+been reported, please go ahead and report it and link to issues you think might
+be related. In general, ICEs on the same line but with different *query stacks*
+are usually distinct bugs.


Maybe a few examples, such as:

panicked at 'index out of bounds: the len is 1 but the index is 1', [...] ena-0.14.1/src/snapshot_vec.rs:199:10

There are other common ones that maybe @matthiaskrgr remembers.

Added an example!

A couple more common ICEs

7 "error_reason": "thread 'rustc' panicked at 'Box<dyn Any>', /rustc/1203e0866e6c3659775efcb8aecad21dc13ef38b/compiler/rustc_errors/src/lib.rs:995:33", 13 "error_reason": "thread 'rustc' panicked at 'Box<dyn Any>', /home/matthias/vcs/github/rust_debug_assertions/compiler/rustc_errors/src/lib.rs:1644:9", 14 "error_reason": "thread 'rustc' panicked at 'forcing query with already existing `DepNode`", 20 "error_reason": "error: internal compiler error: compiler/rustc_infer/src/infer/lexical_region_resolve/mod.rs:203:17: cannot relate region: ReErased", 37 "error_reason": "error: internal compiler error: no errors encountered even though `delay_span_bug` issued", 80 "error_reason": "thread 'rustc' panicked at 'assertion failed: `(left == right)`", 91 "error_reason": "error: internal compiler error: unexpected panic",

compiler-errors · 2023-03-16T04:26:19Z

src/fuzzing.md

+Here are a few things you can do to help the Rust project after filing an ICE.
+
+- Add the minimal test case to [Glacier][glacier]
+- [Bisect][bisect] the bug to figure out when it was introduced 


✨

If this list is arbitrarily ordered, I would prefer if we reorder this in terms of value to the ICE solver / compiler person like me 😸 :

Bisecting

This one is extremely important when it comes to triage. If an ICE is a stable-to-beta regression, it's a far better candidate for immediate fixing. Also, finding the PR that caused the ICE typically makes fixing the ICE significantly easier.

Minimization and fixing unrelated problems feel like the same the same thing, though if you are trying to distinguish them here for some reason, feel free correct me :P

Glacier (these are only super useful for ICEs we expect to be there for a long time)

Minimization and fixing unrelated problems feel like the same the same thing, though if you are trying to distinguish them here for some reason, feel free correct me :P

My only thought here is that it's possible to shink the test case in terms of bytes, LoC, etc. without reducing the number of unrelated errors (indeed, while increasing them).

src/fuzzing.md

langston-barrett · 2023-03-16T12:59:41Z

Github really doesn't want me to re-request review from two people right now... but all reviews are appreciated ofc!

src/fuzzing.md

Noratrieb · 2023-03-16T21:39:24Z

src/fuzzing.md

+- Include all of the information requested in the bug report template
+- Search for existing reports with the same message and query stack
+- Format the test case with `rustfmt`, if it maintains the bug
+- Indicate that the bug was found by fuzzing


I would like to add something like "remove distractions". Often, fuzzed snippets are full of weird code and one of (or even multiple together) these weirdnesses is triggering the ICE. While it's not always easy without understanding the issue, at least trying to remove as many weirdnesses as possible for a few minutes can be great in making the report more obvious.
This includes things like malformed attributes, unresolved names, invalid types etc.

I see you already mentioned something like this below, but I really like the "distractions" wording, Generally, a little less minimal but less "weird" example is easier to understand.

This is discussed a bit later on, under "extra credit" and also under "minimization".

My inclination is to avoid putting too many things in the "Please do" list to make sure that people don't see it and think either (1) "That's so much work! I'm not going to do all that, I'm just going to report it however I feel" or (2) "That's so much work! I'm going to fuzz something else.". I think we want a short list of guidelines that include everything at the "just right" intersection of important and easy.

I put bisection (even though it's important), minimization, and adding to Glacier under "extra credit", because they seem a bit harder. I put rustfmt in this list because it's so easy. Removing spurrious other errors also seems a bit difficult, or at least something I wouldn't necessarily want to do for dozens of bugs.

All of that being said! I've never fixed a bug in rustc, so I don't have a good picture of what's most important.

I see you already mentioned something like this below, but I really like the "distractions" wording, Generally, a little less minimal but less "weird" example is easier to understand.

I clarified this point in the "extra credit" section and used this wording!

langston-barrett · 2023-03-16T22:42:00Z

It occurs to me that there's a lot in this chapter that applies to ICEs in general (though much of it applies especially to fuzzed ICEs), such as requesting minimization, avoiding "distracting" errors, searching for duplicates, bisection, etc. I wonder if it might be helpful to have a separate chapter to hold some of this information. Such a chapter could then be linked to from this one, and maybe also in the ICE issue template.

src/fuzzing.md

langston-barrett · 2023-03-16T23:21:13Z

src/fuzzing.md

+
+[bisect]: https://github.com/rust-lang/cargo-bisect-rustc/blob/master/TUTORIAL.md
+
+## Minimization


I just found two Zulip threads about minimization:

https://rust-lang.zulipchat.com/#narrow/stream/122651-general/topic/minimization/near/342432199

https://rust-lang.zulipchat.com/#narrow/stream/122651-general/topic/Script.20for.20minimizing.20huge.20minimal.20examples/near/266972944

One of them mentions rust-reduce, I will give that tool a try. If successful, I'll add it to the list.

(Might not do this before merging, but I will try to get to it at some point!)

workingjubilee · 2023-03-16T23:24:25Z

It occurs to me that there's a lot in this chapter that applies to ICEs in general (though much of it applies especially to fuzzed ICEs), such as requesting minimization, avoiding "distracting" errors, searching for duplicates, bisection, etc. I wonder if it might be helpful to have a separate chapter to hold some of this information. Such a chapter could then be linked to from this one, and maybe also in the ICE issue template.

Hmm. I think this guide is getting pretty close to "good as-is", and I'd be fine to land this as one chapter and then do a split in a followup that reorganizes things slightly and cleanly divides "minimization as relevant to fuzzing" out.

langston-barrett · 2023-03-16T23:47:17Z

Hmm. I think this guide is getting pretty close to "good as-is", and I'd be fine to land this as one chapter and then do a split in a followup that reorganizes things slightly and cleanly divides "minimization as relevant to fuzzing" out.

Agreed! Someone on Zulip also mentioned minimization as a potential chapter topic.

compiler-errors

I'm happy to merge this now, then.

Update books ## rust-lang/nomicon 1 commits in 1f3e4cd4fd88b5b5d45feb86a11b6d2f93e5a974..b5f7500fc40775096c2bbd204eae096612cf9047 2023-03-27 13:47:36 UTC to 2023-03-27 13:47:36 UTC - Fix typo in 3.8 Subtyping and Variance (rust-lang/nomicon#395) ## rust-lang/reference 4 commits in 24c87f6663aed55b05d2cc286878f28f21918825..3c47807a3131b3c7cacb508f52632078d253cd0a 2023-03-26 18:42:43 UTC to 2023-03-14 18:28:23 UTC - Relax ordering rules for `asm!` operands (rust-lang/reference#1323) - Improve labeled blocks documentation (rust-lang/reference#1342) - Inline assembly: Fix repeated and unordered items in guaranteed directives (rust-lang/reference#1341) - Clarify that free constants are always evaluated at compile time (rust-lang/reference#1328) ## rust-lang/rust-by-example 7 commits in af0998b7473839ca75563ba3d3e7fd0160bef235..cfbfd648ce33926c3490f24de9a5b56cce404b88 2023-03-21 12:05:17 UTC to 2023-03-21 11:58:20 UTC - Fix two typos in the asm chapter (rust-lang/rust-by-example#1692) - Change `runtime` error to `compile time` error (rust-lang/rust-by-example#1690) - Fix tests running on non-x86 platforms. (rust-lang/rust-by-example#1687) - Remove trailing semicolon from macro expression (rust-lang/rust-by-example#1683) - Explained why it should not work in Chapter 4.1 (rust-lang/rust-by-example#1682) - Fix comment to mention the correct type of error (rust-lang/rust-by-example#1680) - Improve the content for `read_lines` (rust-lang/rust-by-example#1679) ## rust-lang/rustc-dev-guide 9 commits in b1b6d693cd1461e53de4132c1b183ace31cd36e5..d08baa166b463537229eeb737c4ccadabd83cf78 2023-03-26 17:55:53 UTC to 2023-03-14 03:50:20 UTC - Add locale_resources (rust-lang/rustc-dev-guide#1651) - Don't require $GITHUB_TOKEN to build locally (rust-lang/rustc-dev-guide#1652) - bootsrapping stages overview list (rust-lang/rustc-dev-guide#1555) - Update labels overview (rust-lang/rustc-dev-guide#1639) - first mention of type, and add a link (rust-lang/rustc-dev-guide#1643) - Add SIP solution for macOS users (rust-lang/rustc-dev-guide#1636) - Add chapter on fuzzing (rust-lang/rustc-dev-guide#1646) - Fix "Crate disambiguator" in libs-and-metadata.md (rust-lang/rustc-dev-guide#1648) - Update rustdoc-internals.md (rust-lang/rustc-dev-guide#1644)

Add chapter on fuzzing

7a8e791

compiler-errors reviewed Mar 16, 2023

View reviewed changes

workingjubilee reviewed Mar 16, 2023

View reviewed changes

src/fuzzing.md Outdated Show resolved Hide resolved

src/fuzzing.md Outdated Show resolved Hide resolved

src/fuzzing.md Outdated Show resolved Hide resolved

Address review comments

48864ab

langston-barrett requested review from compiler-errors and workingjubilee and removed request for compiler-errors and workingjubilee March 16, 2023 12:58

mir-opt*-level*, not mir-opt

b941d22

matthiaskrgr reviewed Mar 16, 2023

View reviewed changes

src/fuzzing.md Show resolved Hide resolved

matthiaskrgr reviewed Mar 16, 2023

View reviewed changes

src/fuzzing.md Outdated Show resolved Hide resolved

src/fuzzing.md Outdated Show resolved Hide resolved

src/fuzzing.md Outdated Show resolved Hide resolved

src/fuzzing.md Outdated Show resolved Hide resolved

langston-barrett added 3 commits March 16, 2023 16:56

Address review comments

7e6c2ea

Mention debug assertions

80b04f9

Mention debug assertions label

19cafc0

Noratrieb reviewed Mar 16, 2023

View reviewed changes

Reword to include 'distractions'

a38cd10

langston-barrett commented Mar 16, 2023

View reviewed changes

src/fuzzing.md Show resolved Hide resolved

langston-barrett commented Mar 16, 2023

View reviewed changes

compiler-errors approved these changes Mar 17, 2023

View reviewed changes

compiler-errors merged commit 7cef03a into rust-lang:master Mar 17, 2023

langston-barrett deleted the fuzzing branch March 17, 2023 01:42

rustbot mentioned this pull request Mar 27, 2023

Update books rust-lang/rust#109669

Merged

workingjubilee mentioned this pull request Jul 11, 2023

ICE: rustdoc json when #[doc(primitive)] mod name is same as primitive name. rust-lang/rust#102363

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add chapter on fuzzing #1646

Add chapter on fuzzing #1646

langston-barrett commented Mar 16, 2023

compiler-errors Mar 16, 2023

langston-barrett Mar 16, 2023

matthiaskrgr Mar 16, 2023

compiler-errors Mar 16, 2023 •

edited

Loading

langston-barrett Mar 16, 2023

langston-barrett commented Mar 16, 2023

Noratrieb Mar 16, 2023

Noratrieb Mar 16, 2023

langston-barrett Mar 16, 2023

langston-barrett Mar 16, 2023

langston-barrett commented Mar 16, 2023

langston-barrett Mar 16, 2023

langston-barrett Mar 17, 2023

workingjubilee commented Mar 16, 2023

langston-barrett commented Mar 16, 2023

compiler-errors left a comment


		[bisect]: https://github.com/rust-lang/cargo-bisect-rustc/blob/master/TUTORIAL.md

		## Minimization

Add chapter on fuzzing #1646

Add chapter on fuzzing #1646

Conversation

langston-barrett commented Mar 16, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

compiler-errors Mar 16, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

langston-barrett commented Mar 16, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

langston-barrett commented Mar 16, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

workingjubilee commented Mar 16, 2023

langston-barrett commented Mar 16, 2023

compiler-errors left a comment

Choose a reason for hiding this comment

compiler-errors Mar 16, 2023 •

edited

Loading