-
Notifications
You must be signed in to change notification settings - Fork 586
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Update the GATK base image to a newer LTS ubuntu release (#8610)
* Update the GATK base image to the latest Ubuntu LTS release (22.04) * Add some additional useful utilities to the base image * Switch to a newer conda version with a much faster solver * Update the scripts and documentation for building the base image * Update the VETS integration tests to allow for a small epsilon during numeric comparisons, and include the full diff output in exception messages when a mismatch is detected
- Loading branch information
Showing
14 changed files
with
298 additions
and
153 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,16 +1,15 @@ | ||
| # How to update the GATK base docker image: | ||
|
|
||
| 1. choose a new version number for the base image and manually update the version in `scripts/docker/gatkbase/build_docker_base.sh` | ||
| 2. build the gatkbase image using that script and upload it to the [gatk-dev docker repo](https://hub.docker.com/r/broadinstitute/gatk-dev/) or [gcr-gatk-snapshots](us.gcr.io/broad-dsde-methods/broad-gatk-snapshots) | ||
| * cd to scripts/docker/gatkbase | ||
| * run `./build_docker_base.sh` | ||
| * `docker tag broadinstitute/gatk-dev:your-version-rc1` or whatever the correct tag is for where you want it uploaded | ||
| * `docker push tagname` | ||
| 3. update the Dockerfile in the main gatk to use the image you pushed | ||
| 4. commit the changes to the two docker files and to a new pull request | ||
| 5. wait for the tests to pass and show it to a reviewer | ||
| 6. push the base image to the official [gatk repo](https://hub.docker.com/r/broadinstitute/gatk) with the right name | ||
| 7. update the main docker to point to the official version you just released | ||
| 8. wait for tests to pass in travis and merge | ||
| 1. In a branch, make whatever updates you need to the base image Dockerfile in `scripts/docker/gatkbase` | ||
| 2. Choose a new version number for the new base image | ||
| 3. Build the GATK base image by running either `build_docker_base_cloud.sh` (to build in the cloud using Google Cloud Build) or `build_docker_base_locally.sh` (to build on your local machine) from within the `scripts/docker/gatkbase` directory in your GATK clone. | ||
| * Both scripts take the version of the new image as their only argument. Eg., `build_docker_base_cloud.sh 3.0.0rc1` will create the remote image `us.gcr.io/broad-dsde-methods/gatk-base-image-staging-area:gatkbase-3.0.0rc1` | ||
| 4. If you built the image locally, you'll need to manually push it to the staging area at `us.gcr.io/broad-dsde-methods/gatk-base-image-staging-area` | ||
| 5. Test the new image using the GATK branch with your changes to the base image Dockerfile, by modifying the FROM clause at the top of the main GATK Dockerfile to point to the base image you staged, and submit a PR for the branch to let the test suite run. | ||
| 6. Once tests pass and everything looks good, push the new base image to the release repositories using the `release_prebuilt_base_image.sh` script | ||
| * The release script takes as arguments the prebuilt image you've tested, and a final version number for release. For example: `release_prebuilt_base_image.sh us.gcr.io/broad-dsde-methods/gatk-base-image-staging-area:gatkbase-3.0.0rc1 3.0.0` | ||
| * The release script looks for the image locally first, and if not found locally it pulls it from the remote repository before releasing. | ||
| 7. Update the FROM clause in the main GATK Dockerfile in your GATK PR to point to the officially-released image, and merge it once tests pass. | ||
|
|
||
|
|
||
|
|
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| #!/bin/bash | ||
| # | ||
| # A script that builds the GATK base image using Google Cloud Build, and pushes it to | ||
| # a staging location at us.gcr.io/broad-dsde-methods/gatk-base-image-staging-area | ||
| # | ||
| # Usage: build_docker_base_cloud.sh <docker_image_version> | ||
| # | ||
| # After staging the image, you should test it with GATK before actually releasing it | ||
| # using the release_prebuilt_base_image.sh script. You can test it by modifying the | ||
| # main GATK Dockerfile FROM clause to point temporarily at the staged base image, and | ||
| # submitting that as a PR to trigger a test suite run. | ||
| # | ||
|
|
||
| if [ $# -ne 1 ]; then | ||
| echo "Usage: $0 <docker_image_version>" | ||
| exit 1 | ||
| fi | ||
|
|
||
| IMAGE_VERSION=$1 | ||
| IMAGE_NAME="us.gcr.io/broad-dsde-methods/gatk-base-image-staging-area" | ||
| DOCKER_IMAGE_TAG="${IMAGE_NAME}:gatkbase-${IMAGE_VERSION}" | ||
|
|
||
| gcloud builds submit --tag ${DOCKER_IMAGE_TAG} --timeout=24h --machine-type n1_highcpu_32 | ||
|
|
||
| if [ $? -ne 0 ]; then | ||
| echo "gcloud builds submit failed" | ||
| exit 1 | ||
| fi | ||
|
|
||
| echo "Successfully published image to staging area at ${DOCKER_IMAGE_TAG}" | ||
| exit 0 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| #!/bin/bash | ||
| # | ||
| # A script that builds the GATK base image locally (but does not push it anywhere). | ||
| # | ||
| # Usage: build_docker_base_locally.sh <docker_image_version> | ||
| # | ||
| # After building the image, you should test it with GATK before actually releasing it | ||
| # using the release_prebuilt_base_image.sh script. You can test it by modifying the | ||
| # main GATK Dockerfile FROM clause to point temporarily at the candidate base image, and | ||
| # submitting that as a PR to trigger a test suite run. | ||
| # | ||
|
|
||
| if [ $# -ne 1 ]; then | ||
| echo "Usage: $0 <docker_image_version>" | ||
| exit 1 | ||
| fi | ||
|
|
||
| IMAGE_VERSION=$1 | ||
| IMAGE_REPO="us.gcr.io/broad-dsde-methods/gatk-base-image-staging-area" | ||
| IMAGE_FULL_TAG="${IMAGE_REPO}:gatkbase-${IMAGE_VERSION}" | ||
|
|
||
| # Build | ||
| echo "Building image to tag ${IMAGE_FULL_TAG}..." | ||
| docker build --squash -t "${IMAGE_FULL_TAG}" . | ||
|
|
||
| if [ $? -ne 0 ]; then | ||
| echo "docker build failed" | ||
| exit 1 | ||
| fi | ||
|
|
||
| echo "docker build succeeded" | ||
| exit 0 | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,60 @@ | ||
| #!/bin/bash | ||
| # | ||
| # A script that takes a prebuilt GATK base image, and pushes it to the GATK release repositories on | ||
| # dockerhub and GCR. Use the build_docker_base_cloud.sh or build_docker_base_locally.sh scripts to | ||
| # build the image before running this script. | ||
| # | ||
| # Usage: release_prebuilt_base_image.sh <prebuilt_image> <version_number_for_release> | ||
| # | ||
| # If the prebuilt image exists locally, this script will push the local version. Otherwise, | ||
| # it will pull the image from the remote repository before pushing it to the GATK release | ||
| # repositories. | ||
| # | ||
| # prebuilt_image: The pre-built image you want to release (make sure you've tested it!) | ||
| # May be either local or remote | ||
| # version_tag_for_release: The version of the GATK base image you're releasing (eg., 3.1.0) | ||
| # | ||
|
|
||
| if [ $# -ne 2 ]; then | ||
| echo "Usage: $0 <prebuilt_image> <version_tag_for_release>" | ||
| exit 1 | ||
| fi | ||
|
|
||
| PREBUILT_IMAGE="$1" | ||
| VERSION="$2" | ||
| DOCKERHUB_REPO="broadinstitute/gatk" | ||
| GCR_REPO="us.gcr.io/broad-gatk/gatk" | ||
| BASE_IMAGE_FULL_TAG="gatkbase-${VERSION}" | ||
|
|
||
| function fatal_error() { | ||
| echo "$1" 1>&2 | ||
| exit 1 | ||
| } | ||
|
|
||
| function docker_push() { | ||
| echo "Pushing to ${1}" | ||
| docker push "${1}" | ||
| if [ $? -ne 0 ]; then | ||
| fatal_error "Failed to push to ${1}" | ||
| fi | ||
| } | ||
|
|
||
| # Test if the prebuilt image exists locally, and pull it if it doesn't | ||
| docker image inspect "${PREBUILT_IMAGE}" > /dev/null 2>&1 | ||
| if [ $? -ne 0 ]; then | ||
| echo "Image ${PREBUILT_IMAGE} not found locally: attempting to pull it now" | ||
| docker pull "${PREBUILT_IMAGE}" | ||
| if [ $? -ne 0 ]; then | ||
| fatal_error "Failed to pull pre-built image ${PREBUILT_IMAGE}" | ||
| fi | ||
| else | ||
| echo "Image ${PREBUILT_IMAGE} found locally: pushing it to the release repositories" | ||
| fi | ||
|
|
||
| docker tag "${PREBUILT_IMAGE}" "${DOCKERHUB_REPO}:${BASE_IMAGE_FULL_TAG}" | ||
| docker tag "${PREBUILT_IMAGE}" "${GCR_REPO}:${BASE_IMAGE_FULL_TAG}" | ||
|
|
||
| docker_push "${DOCKERHUB_REPO}:${BASE_IMAGE_FULL_TAG}" | ||
| docker_push "${GCR_REPO}:${BASE_IMAGE_FULL_TAG}" | ||
|
|
||
| exit 0 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.