Add splitting algorithm #16
Conversation
|
For my testing suite: |
|
So the new grouping is more balanced, although the difference is not massive. |
|
@jerry-git @sondrelg please review |
|
Links back to this discussion: #14 (comment) |
| * Absolute Order: whether each group contains all tests between first and last element in the same order as the original list of tests | ||
| * Relative Order: whether each test in each group has the same relative order to its neighbours in the group as in the original list of tests | ||
|
|
||
| The `duration_based_chunks` algorithm aims to find optimal boundaries for the list of tests and every test group contains all tests between the start and end bounary. |
There was a problem hiding this comment.
I think it could be valuable to mention also in the usage section that one can specify the splitting algorithm via command line arg and also mention what is the default behaviour
src/pytest_split/algorithms.py
Outdated
| from _pytest import nodes | ||
|
|
||
|
|
||
| ALGORITHMS = ["duration_based_chunks", "least_duration"] |
There was a problem hiding this comment.
I think enum could make more sense here
There was a problem hiding this comment.
Changed to enum. Please let me know if I made the change you intended.
|
|
||
| class TestAlgorithms: | ||
| @pytest.mark.parametrize("algo_name", algorithms.ALGORITHMS) | ||
| def test__split_test(self, algo_name): |
There was a problem hiding this comment.
Let's keep consistency, other test modules seem to use single _ after test 🙂
There was a problem hiding this comment.
It is consistent in the sense that I use the format: test_{func}_{does_something}_{when}.
And in this case the func is called _split_test.
Please let me know if you still prefer the change.
|
@jerry-git I don't understand why the CI fails. It seems like the module is not actually installed? |
|
Hmm, it's interesting that it works for |
|
I had the same problem in the Poetry setup PR but I can't remember how I fixed it. I wonder if you should just remove the |
|
Yeah I was considering removing |
|
Which ever it is, the tests running in this repo seem to pass and the tests running in the fork fail, right I can't think of a reason why this would happen, but there might be a subtle difference between |
The new splitting algorithm goes over the items, determines a best estimate for the duration and then adds it to the smallest group. The are edge-cases that this algorithm can't handle properly. For example, when the last test item has a large test duration, the current solution won't create the optimal solution. This can be improved by sorting the test durations first, and starting with assignment of tests that have the largest test duration.
The old algorithm was restored, and the new one was added as well. The user can now choose which algorithm is most suitable for their use case.
|
thanks @jerry-git , the tests succeed now. |
This makes two updates:
I'll share some anecdotal information about how the new algorithm performs vs the old.