very cool!

[matthiaskrgr]

Hey, just wanted to say hi! :D
Very cool crate!
I have hooked this up into icemaker to automatically generate code from a set of input which then can be checked (I already tried something similar previously by feeding everything into a markov chain (lol) and looking what pops out.

Two small things I noticed:
It might be nice if the splice function would not take the hashmap by reference so that it is possible to have a function that takes a path, creates the hashmap, feeds it into splice and then returns an Iterator , right now I think the iterator would outlive the ref to the HashMap and thus it would not work (I've worked around with a Vec right now, meh...)

I have not gotten to debug this in detail but when I processed rustc tests/ui/issues/issue-74564-if-expr-stack-overflow.rs I ran into a stack overflow. (yay, for fuzzing the fuzzer!), but I could not reproduce this when running the splicer binary directly on it so far...

[langston-barrett]

Thanks for reaching out! Glad you think it's cool. You might also be interested in treereduce, which can be used to minimize the bugs that get found.

I have hooked this up into icemaker to automatically generate code from a set of input which then can be checked (I already tried something similar previously by feeding everything into a markov chain (lol) and looking what pops out.

I was thinking of doing this 😄 I would appreciate it if you could add a shout-out in the README and/or notes when you find bugs using it! But no worries either way, I don't really care that much.

I think it would be so cool if we could get an integrated, end-to-end solution with icemaker, tree-splicer, and treereduce 🚀 Maybe it would even write the bug description for you?

It might be nice if the splice function would not take the hashmap by reference so that it is possible to have a function that takes a path, creates the hashmap, feeds it into splice and then returns an Iterator , right now I think the iterator would outlive the ref to the HashMap and thus it would not work (I've worked around with a Vec right now, meh...)

This is a bit tricky... I want to share references between the Splicer and the Branches, but I can't do that if Splicer holds the HashMap. I'm open to suggestions and PRs, though!

I have not gotten to debug this in detail but when I processed rustc tests/ui/issues/issue-74564-if-expr-stack-overflow.rs I ran into a stack overflow. (yay, for fuzzing the fuzzer!), but I could not reproduce this when running the splicer binary directly on it so far...

I'm not surprised, haha. Keep me posted if you're able to track it down!

[matthiaskrgr]

Added a small section to the readme, let me know what you think :)
matthiaskrgr/icemaker@6ad6238

rust-lang/rust#109071
rust-lang/rust#109072
rust-lang/rust#109090
emerged from tree-splicer gen'd code, but so far I have only touched the tip of the iceberg I think (I have checked perhaps 700 MB of newly generated code, I still have ~19 gigabyte of already-checked code (more or less syntactically valid.. 😅 ) that I can use to generate new code with... :)

Filling out a bug template with input code, used compiler flags, stacktrace and compiler version should be pretty easy as I already have all this information available (icemaker produces a logfile which looks somewhat like this: https://github.com/matthiaskrgr/arctICE/blob/main/errors.json

Automatic testcase reduction is definitely something I have been thinking about a lot already but it would require some kind of directory to dump the reduced files to. Maybe I will get to that in the upcoming weeks...

[langston-barrett]

Added a small section to the readme, let me know what you think :)

Thanks :)

(more or less syntactically valid.. sweat_smile )

That's one of the great tihngs about tree-sitter - it can deal with partially-invalid code!

rust-lang/rust#109071
rust-lang/rust#109072
rust-lang/rust#109090
emerged from tree-splicer gen'd code, but so far I have only touched the tip of the iceberg I think (I have checked perhaps 700 MB of newly generated code, I still have ~19 gigabyte of already-checked code (more or less syntactically valid.. sweat_smile ) that I can use to generate new code with... :)

Awesome!! I'll add them to the README.

Automatic testcase reduction is definitely something I have been thinking about a lot already but it would require some kind of directory to dump the reduced files to. Maybe I will get to that in the upcoming weeks...

Feel free to ping me if I can be helpful on this front, with API adjustments to treereduce or anything else.

[matthiaskrgr]

I also had a look at treereduce now, but it took me a while until I had the flags figured out..
treereduce-rust --slow --source a.rs --interesting-exit-code 101 -- rustc -Zflags-go-here @@.rs especially the "@@" stuff 😅

from the --help output Usage: treereduce-rust <CMD>... I thought I could perhaps use treereduce-rust "rustc file.rs -Zflags" but this did not work 🙃

It might also be nice if we could pass a regex like --interesting-output="panicked at .*" which the makes treereduce check stdout and stderr for this regex and if it matches it knows the program still crashes.

Oh and it could run rustfmt on the final file (before making sure that does not "fix" the crash of course)

[langston-barrett]

I also had a look at treereduce now, but it took me a while until I had the flags figured out..
treereduce-rust --slow --source a.rs --interesting-exit-code 101 -- rustc -Zflags-go-here @@.rs especially the "@@" stuff sweat_smile

Yeah, it's a little confusing, sorry! The docs have an example: https://langston-barrett.github.io/treereduce/usage.html

It might also be nice if we could pass a regex like --interesting-output="panicked at .*" which the makes treereduce check stdout and stderr for this regex and if it matches it knows the program still crashes.

My thoughts exactly! langston-barrett/treereduce#6 langston-barrett/treereduce#45

For now, you can give it a shell script that uses grep (grep outputs 0 if it finds a match, 1 otherwise).

Oh and it could run rustfmt on the final file (before making sure that does not "fix" the crash of course)

I think this would be great in a combined tool.

[matthiaskrgr]

🧊
rust-lang/rust#109148
rust-lang/rust#109147
rust-lang/rust#109146
rust-lang/rust#109145
rust-lang/rust#109144
rust-lang/rust#109141

[langston-barrett]

@matthiaskrgr Here's a wrapper around treereduce-rust that just fills in a few command-line arguments to make it easier to reduce ICEs specifically: https://github.com/langston-barrett/icemelter

This weekend, I'll improve it so that it avoids introducing spurrious syntax/type/scope/borrow errors.

[matthiaskrgr]

Oh that's interesting, I'll check it out :)
I have have been thinking about this a bit.
If we have special rustcflags required to repro the crash, we could in theory add two conditions for "successfull minimization" which is

1. the code must crash with the flag and
2. the code must still compile without the flag (no errors)

[langston-barrett]

It now avoids introducing spurrious errors, at the cost of sometimes not being able to reduce a problem at all.

[matthiaskrgr]

https://github.com/rust-lang/rust/issues/109305  
https://github.com/rust-lang/rust/issues/109304  
https://github.com/rust-lang/rust/issues/109300  
https://github.com/rust-lang/rust/issues/109299  
https://github.com/rust-lang/rust/issues/109298  
https://github.com/rust-lang/rust/issues/109297  
https://github.com/rust-lang/rust/issues/109296  
https://github.com/rust-lang/rustfmt/issues/5716  
https://github.com/rust-lang/rust/issues/109239  
https://github.com/rust-lang/rust/issues/109232  
https://github.com/rust-lang/rust/issues/109204  
https://github.com/rust-lang/rust/issues/109191  
https://github.com/rust-lang/rust/issues/109188  
https://github.com/rust-lang/rust/issues/109178  
https://github.com/rust-lang/rust/issues/109152  

[langston-barrett]

@matthiaskrgr A few updates:

• tree-splicer v0.3.0 can now also delete optional subtrees at random, it can now "stack" splices (splice into a splice), and it introduces "chaotic" mutations that may be syntactically invalid but can help introduce even more variety
• icemelter v0.3.0 can now run rustfmt and cargo-bisect-rustc for you automatically