Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement is_minimal_case #477

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

Conversation

Andlon
Copy link
Contributor

@Andlon Andlon commented Jul 5, 2024

This is a draft for an implementation of the idea suggested in #161.

Currently, using proptest for debugging is very cumbersome because you cannot easily run code only for the minimal failing test case. In my experience, I usually only care about the minimal test case, but if I want to print some debug info about that failing test case, I'll have to print for all test cases, which is borderline useless.

This PR introduces a method is_minimal_case that you can call inside a proptest to determine if the current test case is the minimal failing case. To do this, the minimal test case is run one more time after it has been identified, and a thread local boolean is set before and after.

This allows you to write tests like these:

use proptest::{proptest, prop_assert, is_minimal_case};

proptest! {
    #[test]
    fn test_is_not_five(num in 0 .. 10) {
        if is_minimal_case() {
            eprintln!("Minimal test case is {num:?}");
            export_to_file_for_analysis(num);
        }

        prop_assert!(num != 5);
    }
}

A nice side effect of this is that it makes it much easier to use proptest with a debugger: Simple add an if condition like the one above, and put a breakpoint on a statement inside the branch. Then you can run the debugger only through the minimal failing test case.

I haven't yet written any tests, because I'm frankly not quite sure how best to go about it. I suppose at a minimum we should have a test where a TestRunner is manually invoked with a dummy test case, where we test that the reported minimal test case is what we expect. But I'm not sure how best to write such a test in a way that fits well with the rest of the code base. I'd be very grateful to have some suggestions here.

I'm not sure how to correctly handle forking here - if anything in particular even needs to be done differently. And whether or not I should forward is_from_persistent_seed. Since I'm not so familiar with the code base I don't really understand the ramifications.

Happy to make any necessary adjustments to the PR.

@Andlon
Copy link
Contributor Author

Andlon commented Jul 6, 2024

I have no idea what happened with CI. It seems that it just hangs on some build script?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant