Tests on CI run slow, too often and flaky

[usu]

This is a collection of issues I see with our current CI setup that make development slow and makes the life of renovate hard.

Problem description

• Generally, our tests take a rather long time (~15min in average)
• The tests run often, sometimes testing the same commit that was already tested before
• Our e2e tests are still flaky, leading to necessary repetition of tests
  • Higher utilization of workflow runners
  • PRs getting kicked out of merge queue (=queue has to restart other jobs)
  • Renovate not being able auto-merge => high clutter in PR backlog
• Fixing a bug and deploying it to prod takes a huge time (as we saw during the last hackathon). CI needs to run min. 6 times successfully:
  • First, CI needs to run on PR (twice in parallel)
  • Then CI needs to run on merge queue
  • After merging a PR from merge queue, dev again runs CI (although the same commit was already tested in merge queue)
  • When fast-forwarding to stage & prod, the CI runs again (although the same commit was already tested before)
• Renovate still seems to have a problem with our setup: Renovate mixing different upgrades in same PR and even downgrades packages #4309
  This might be unrelated to above. I couldn't figure out anything on renovatebot issues list. I wonder though if this is somehow connected to merge queue and the config rebaseWhen:conflicted.

Potential solutions (to be discussed)

• Challenge benefit of merge queue (does it bring the benefits we hoped?)
• Making e2e tests non-mandatory tests? Or at least for renovate? Yes, this means we could potentially break dev. I don't think though that I remember a dependency upgrade only breaking e2e and no other tests.
• Somehow making API tests faster
• When pushing to a pull request, the CI runs twice (1x for push, 1x for pull request). Do we really need both?
• Find a way to avoid running the same CI for a commit hash where CI has already run successfully before
• other ideas?

[manuelmeister]

Challenge benefit of merge queue (does it bring the benefits we hoped?)

I quite like it. Especially when many people are actively pushing/merging

Making e2e tests non-mandatory tests? Or at least for renovate? Yes, this means we could potentially break dev. I don't think though that I remember a dependency upgrade only breaking e2e and no other tests.

I think we should try to make the e2e test less flakey. I don't like not testing them, as before we upgrade to vue3 I'd like add more tests. We could try to use https://docs.cypress.io/guides/continuous-integration/github-actions#Testing-with-Cypress-Docker-Images instead of pulling the latest one.

Somehow making API tests faster

The question is, are we testing the right way?

When pushing to a pull request, the CI runs twice (1x for push, 1x for pull request). Do we really need both?

That was once the plan.

Find a way to avoid running the same CI for a commit hash where CI has already run successfully before

We could try to run e.g. the api test on PR push only if certain conditions match?

[manuelmeister]

Core Meeting Decision

Things we want to tackle:

• Try to use cypress image CI: use image in docker compose again to run e2e tests #5414
• Improve Stability of cypress tasks
• Improve Stability of api tests (fix api performance_test #5292)
• Reduce duration of api tests (maybe parallelism)
  • Happened without our doing: Destroy TestCase object after its test was run sebastianbergmann/phpunit#5861
  • Further reduction of CI time: Use paratest for parallel test execution #5372
• Increase the duration of the performance check. We would like to have a benchmark, because the CI is not guaranteed to run stable or performant
  • Create reporting of certain dev endpoints under repeatable circumstances
• Local E2E use separate DB

Maybe/explore:

• Only run stable e2e tests
• skip duplicate actions