-
Notifications
You must be signed in to change notification settings - Fork 29
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Splits invalid when collection order not deterministic #25
Comments
I assume it's not deterministic here because of |
No, this problem occurs when either the data structure has non-deterministic order or the code generating the parametrised test cases is for some reason not deterministic. |
I think we could go with OTOH, maybe it's better to make sure that we don't accidentally skip tests (or run some test in multiple groups) 🤔 With these thoughts, I'd go with |
Yes, and I wonder if we should perhaps be safe by default (i.e. option 1) and allow users to do the unsafe thing (existing behaviour). If we make clear what the tradeoffs are, the user can then decide for him/herself. In other words add a parameter that |
Was this implemented yet? If not I think it would be a great feature to have, as I have some tests I want ran in the same group under the same parametrize decorator |
The big assumption underlying the two splitting algorithms is that the order of collected items is constant.
However, I've come across a case where this assumption was violated.
In my case I had a test parametrised with
pytest.mark.parametrize
, but the items to parametrize with would sometimes change order.Take this example:
If you run this often enough you'll see that the order changes:
and
I'm not sure how to address this, but I think there are a few options:
parametrize
for the same test. In other words, make sure that a single group will run all tests fortest_hello
.pytest
with those pre-calculated groups (so not really using this plugin as a plugin :p)The text was updated successfully, but these errors were encountered: