-
Notifications
You must be signed in to change notification settings - Fork 180
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Existing sample/recipe for integrating asv with github #796
Comments
@yarikoptic did you manage to build such a service? |
Nope, but my colleagues set it up to just run asv to get an estimate for performance impact comparing against master in PRs. See e.g. our datalad github actions workflow: https://github.com/datalad/datalad/blob/master/.github/workflows/benchmarks.yml and the actual helper https://github.com/datalad/datalad/blob/master/tools/ci/benchmark-travis-pr.sh . Eventually we just might look into establishing self-hosted runner for actions: https://help.github.com/en/actions/automating-your-workflow-with-github-actions/about-self-hosted-runners so we get more stable estimates |
note - that we nohow report back overall effect, this action is merely for manual investigation if there is a suspicion that PR might affect performance. Would be nice to report back to action state if e.g. any benchmark became significantly slower. we haven't looked into that yet |
In case it helps anyone, I just set up My github action looked like this: benchmark:
name: benchmark
runs-on: ubuntu-latest
steps:
- uses: actions/setup-python@v2
with:
python-version: "3.10"
- uses: actions/checkout@v3
- run: git fetch origin $GITHUB_BASE_REF:base $GITHUB_REF:pr
- run: pip install asv virtualenv
- run: asv machine --yes
- run: asv continuous base pr --strict -e |
in https://github.com/datalad/datalad/blob/master/.github/workflows/benchmarks.yml we also fail the workflow if any benchmarks gets significantly worse: - name: Fail if any benchmarks have slowed down too much
run: |
! asv compare --factor 1.2 --split refs/bm/merge-target refs/bm/pr | grep -q "got worse" |
Backstory aside - is anything aware of an existing use of asv integrated with GitHub?
What I envision is having a container image with eg buildbot and asv installed, to be deployed on a dedicated box, configured to monitor/report back on PRs to GitHub about impact on benchmarks.
May be there is a service which already does something like that?
I will look into setting up such one, but wanted to make sure I am not reinventing the wheel or making it square.
The text was updated successfully, but these errors were encountered: