Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace Travis with GithubActions #7754

Merged
merged 37 commits into from
Apr 21, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
b48f74a
Switched over to using GithubActions for CI services and re-implement…
lbergelson Mar 16, 2022
5bdadc5
removing branch specific changes
jamesemery Apr 8, 2022
df56d68
fixing another directory reference in the cnv somatic wdls
jamesemery Apr 11, 2022
9042312
removing a namespace collision
jamesemery Apr 11, 2022
4e99ce8
changed the order of some things
jamesemery Apr 11, 2022
d099357
testing again with local actions
jamesemery Apr 12, 2022
013ae90
removing test training wheels
jamesemery Apr 12, 2022
fb10249
more local actions
jamesemery Apr 12, 2022
033b9b7
testing some supurflous tests for superflouity
jamesemery Apr 13, 2022
6a01f8e
testing with changed around environment variables
jamesemery Apr 13, 2022
027c38e
docker labeling the artifcats
jamesemery Apr 13, 2022
68ae653
bash expression magic
jamesemery Apr 13, 2022
00b2330
using a ternary expresson workaround
jamesemery Apr 13, 2022
1b5a063
a few final comments, tweaks, and renames before passing it back and …
jamesemery Apr 13, 2022
e1cffc1
testing docker artifacting
jamesemery Apr 14, 2022
1d4afa3
hiring another driver
jamesemery Apr 14, 2022
8a6d40a
another try
jamesemery Apr 14, 2022
307d80c
some cleanup after changing over the docker images
jamesemery Apr 14, 2022
09e15b1
changes that hopefully prevent full scale explosions for fokred PRs
jamesemery Apr 14, 2022
5a2ea13
remember your quotations
jamesemery Apr 14, 2022
f2f1102
env.
jamesemery Apr 14, 2022
7aac557
skipping upload steps if tokens are not populated
jamesemery Apr 15, 2022
42e2538
tokens still exist if no priveledge
jamesemery Apr 15, 2022
5b75c52
refactoring this to be slightly cleaner but still eternally gross
jamesemery Apr 15, 2022
bf930b1
typo
jamesemery Apr 15, 2022
830fef2
typo
jamesemery Apr 15, 2022
37b21d7
_-
jamesemery Apr 15, 2022
b0b318e
_-
jamesemery Apr 15, 2022
238c692
{{{{}}}}
jamesemery Apr 15, 2022
c5a425b
sometimes bash execution happens automatically and sometimes it doesn…
jamesemery Apr 15, 2022
e6ea498
inverting and cleaning
jamesemery Apr 15, 2022
f8ab787
fixing a typo in documentation before merging
jamesemery Apr 20, 2022
36489a7
fixing the build report output link for non-docker tests
jamesemery Apr 20, 2022
d88c867
unifying the test outputs
jamesemery Apr 21, 2022
59f5347
unifying the test outputs
jamesemery Apr 21, 2022
69d7cb0
unifying the test outputs
jamesemery Apr 21, 2022
16ed025
i see that was wrong
jamesemery Apr 21, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions .github/actions/install-cromwell/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
name: 'install-cromwell'
description: 'installs cromwell and womtool locally for jobs that require it (and sets their paths as environment variables)'
inputs:
CROMWELL_VERSION: # id of input
description: 'Cromwell Version to download'
required: true
runs:
using: "composite"
steps:
- name: Set Cromwell env variables
run: |
echo "CROMWELL_JAR=${HOME}/cromwell-${{ inputs.CROMWELL_VERSION }}.jar" >> $GITHUB_ENV;
lbergelson marked this conversation as resolved.
Show resolved Hide resolved
echo "WOMTOOL_JAR=${HOME}/womtool-${{ inputs.CROMWELL_VERSION }}.jar" >> $GITHUB_ENV;
shell: bash
- name: Download Cromwell and Womtools
run: |
wget -q -O $CROMWELL_JAR https://github.com/broadinstitute/cromwell/releases/download/${{ inputs.CROMWELL_VERSION }}/cromwell-${{ inputs.CROMWELL_VERSION }}.jar;
wget -q -O $WOMTOOL_JAR https://github.com/broadinstitute/cromwell/releases/download/${{ inputs.CROMWELL_VERSION }}/womtool-${{ inputs.CROMWELL_VERSION }}.jar;
shell: bash
93 changes: 93 additions & 0 deletions .github/actions/upload-gatk-test-results/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
name: 'upload-gatk-test-results'
description: 'Uploads the results for matrix runs of gatk-tests.yml and maybe comments on failing runs on the PR associated with the test'
inputs:
warnPR:
description: 'If true this will add a comment about the jococo tests to the PR with a link to the failed test'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this skipped by only artifact?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes. So the results are still there but they require a download. I will make a PR to make the commenting/reporting more robust for forked PRs.

required: true
job-matrix-id:
description: 'The job-unique identifier for which test was run (numbered by matrix run)'
required: true
repo-path:
description: 'The job-unique identifier to use for uploading the results of this run into a google bucket'
required: true
is-docker:
description: 'Whether to label the uplaod artifact as coming from a docker test'
required: false
identifier:
description: 'Identifier to use for finding the unique name for jobs in order to determine html logs location (eg. "Java 8 build and test integration")'
required: true
## Secrets and token inputs
repo-token:
description: 'The GITHUB_TOKEN secret used for querying the github api to locate the logs for this run'
required: false
bot-comment-key:
description: 'Key corresponding to the user account to be used for making comments on github about test failures'
required: false
## option to skip all but the artifact uplaod
only-artifact:
description: 'if "true" this will skip any uploading steps that require permissions and only upload the artifact file'
required: false
default: 'false'

runs:
using: "composite"
steps:
- name: Upload test results
if: always()
uses: actions/upload-artifact@v2
with:
# ternary expression syntax is workaround found here https://github.com/actions/runner/issues/409
name: test-results-${{ inputs.is-docker == 'true' && 'docker-' || '' }}${{ matrix.Java }}-${{ matrix.testType }}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be easier to have the calling tasks just pass in the name? That way this doesn't have to be aware of docker/java version/general test structures"?

path: build/reports/tests

- name: Upload to codecov
run: bash <(curl -s https://raw.githubusercontent.com/broadinstitute/codecov-bash-uploader/main/codecov-verified.bash)
shell: bash

- name: Upload Reports
if: ${{ inputs.only-artifact != 'true' }}
id: uploadreports
run: |
gsutil -m cp -z html -z js -z xml -z css -r build/reports/tests gs:/${{ env.HELLBENDER_TEST_LOGS }}${{ inputs.repo-path }}/;
VIEW_URL=https://storage.googleapis.com${{ env.HELLBENDER_TEST_LOGS }}${{ inputs.repo-path }}/tests/test/index.html
echo "See the test report at ${VIEW_URL}";
echo ::set-output name=view_url::"${VIEW_URL}"
shell: bash

# This code is necessary in order to extract the URL for the logs for a particular run. The github context exposes the run_id object
# which corresponds to the unique identifier for an individual workflow execution but it doesn't expose the job_id.id which is necessary
# to extract the logs for failing jobs. The best workaround seems to be this approach (querying the API and parsing the json output).
- name: Extract Log Location
if: ${{ inputs.only-artifact != 'true' }}
id: loghtml
env:
GITHUB_TOKEN: ${{ inputs.repo-token }}
run: |
GITHUB_BASEURL=https://api.github.com
REPO_ID=$(curl --get -Ss -H "Authorization: token ${GITHUB_TOKEN}" -H "Accept: application/vnd.github.v3+json" https://api.github.com/repos/${{ github.repository }} | jq -r .id)
GITHUB_API="/repositories/${REPO_ID}/actions/runs/${{ github.run_id }}/jobs"
eval "$(curl --get -Ss -H "Authorization: token ${GITHUB_TOKEN}" -H "Accept: application/vnd.github.v3+json" "${GITHUB_BASEURL}${GITHUB_API}?per_page=30" \
| jq -r --arg job_name "${{ inputs.identifier }}" '.jobs | map(select(.name == $job_name)) | .[0] | @sh "job_id=\(.id) html_url=\(.html_url)"')"

echo ::set-output name=job_id::"${job_id}"
echo ::set-output name=html_url::"${html_url}"
echo ${job_id}
echo ${html_url}
shell: bash

- name: Updating the PR with failed results
if: ${{ inputs.warnPR == 'true' && inputs.only-artifact != 'true' }}
env:
GITHUB_TOKEN: ${{ inputs.bot-comment-key }}
TEST_TYPE: ${{ matrix.testType }}
GITHUB_EVENT_NUMBER: ${{ github.event.number }}
GITHUB_REPOSITORY: ${{ github.repository }}
JOB_MATRIX_ID: ${{ inputs.job-matrix-id }}
GITHUB_RUN_ID: ${{ github.run_id }}
JDK_VERSION: ${{ matrix.Java }}
ACTIONS_JOB_WEB_URL: https://github.com/${{github.repository}}/actions/runs/${{github.run_id}}
GITHUB_LOGS_URL: ${{ steps.loghtml.outputs.html_url }}
run: |
pip install --user PyGithub;
python scripts/github_actions/Reporter.py ${{ steps.uploadreports.outputs.view_url }};
shell: bash
Loading