This document describes the overall automation strategy for the Quarkus superheroes services and how all the automation behind it works. There is a lot of automation and resource generation that happens "behind the scenes" when changes are pushed to the GitHub repo.
There are 3 GitHub action workflows that run when code is pushed: Basic building and testing, Build and push container images, and Create deploy resources.
The Basic building and testing workflow is a "sanity check". It is required to pass before pull requests can be merged.
It runs whenever code is pushed to the main branch as well as upon any pull requests.
Tip
The workflow can also be triggered manually.
It runs ./mvnw clean verify and ./mvnw clean verify -Pnative on the event-statistics, rest-fights, rest-heroes, rest-villains, rest-narration, grpc-locations, and ui-super-heroes applications on Java 17.
The Build and push container images workflow does pretty much what it sounds like: builds and pushes container images. For JVM images, it builds both amd64 and arm64 images. Multi-arch native images are coming soon.
It only runs on pushes to the main branch after successful completion of the above Basic building and testing workflow.
Tip
The workflow can also be triggered manually.
It consists of 4 jobs:
- Build JVM container images
- Build native container images
- Push application container images
- Create application multi-arch manifests
If any step in any of the jobs fail then the entire workflow fails.
This image is a visual of what the workflow consists of:
This job Builds JVM container images for the event-statistics, rest-fights, rest-heroes, rest-villains, rest-narration, grpc-locations, and ui-super-heroes applications on Java 17 (both amd64 & arm64 platforms) using the Docker Build action.
Each container image created has 2 tags:
java{{java-version}}-latest-amd64java{{java-version}}-latest-arm64
Tip
- Replace
{{java-version}}with the Java version the application was built with (i.e.17).
There are a total of 14 images built (7 applications x 1 JVM version x 2 platforms).
This job runs in parallel with the Build JVM container images job.
The job Builds native executable container images for the event-statistics, rest-fights, rest-heroes, rest-villains, grpc-locations, and ui-super-heroes applications using Mandrel.
Note
The rest-narration application currently doesn't support native compilation
Each container image created has 2 tags:
native-latest-amd64native-latest-arm64
There are a total of 14 images built (7 applications x 2 platforms).
Runs after successful completion of the Build JVM container image and Build native container image jobs.
All the container images created in the Build JVM container image and Build native container image jobs (28 total container images/tags) are pushed to https://quay.io/quarkus-super-heroes.
Runs after successful completion of the Push application container images job and in parallel with the Create UI multi-arch manifests job.
All the application container images for each platform (amd64 & arm64) are combined into manifest lists using the docker manifest command. For example, the java{{java-version}}-latest-amd64 and java{{java-version}}-latest-arm64 tags are combined into a single manifest list with the tag java{{java-version}}-latest.
The Create deploy resources workflow is responsible for generating all of the application resources, described in a later section of this document.
It only runs on pushes to the main branch after successful completion of the Build and push container images workflow.
Tip
The workflow can also be triggered manually.
All generated resources are committed to a new branch and a pull request is opened, which is subsequently approved and merged automatically. The reason for the pull request is because the repository has branch protection rules enabled and can't commit directly to the branch.
The cleanup artifacts job is responsible for cleaning up and deleting all artifacts produced by all the other jobs in the workflow. Many container images are created, stored, and shared amongst all the jobs. This could result in several gigabytes of storage used by each instance of the workflow.
This job will always run regardless of the status of the other jobs in the workflow. It uses the delete-run-artifacts GitHub action to perform its work.
The resources and descriptors in the root deploy directory as well as in each individual project's deploy directory (event-statistics, rest-fights, rest-heroes, rest-villains, rest-narration, grpc-locations, and ui-super-heroes) are used for deploying the entire system or subsets of it into various environments (i.e. Docker compose, OpenShift, Minikube, Kubernetes, etc).
Resources in these directories are generated by the Create deploy resources workflow mentioned in the previous section. Any manual changes made to anything in any of those directories will be overwritten by the workflow upon its next execution.
Kubernetes resources are generated into a deploy/k8s directory, either in the project root directory or in each individual project's directory.
Each Quarkus project (event-statistics, rest-fights, rest-heroes, rest-villains, rest-narration, grpc-locations, ui-super-heroes) uses the Quarkus Kubernetes extension to generate Kubernetes and Knative manifests, the Quarkus Minikube extension to generate Minikube manifests, and the Quarkus OpenShift extension to generate OpenShift manifests.
These extensions generate the manifests needed for the application itself but not for any other services. These extensions can also incorporate additional resources by placing additional resources in each project's src/main/kubernetes directory.
The generate-k8s-resources.sh script loops through all versions of each application (Java version 17, both JVM and native - 8 total versions) and merges the contents of files these extensions generate and places them into each project's deploy/k8s directory as well as the respective files in the root deploy/k8s directory.
The generate-k8s-resources.sh script additionally creates the monitoring (Prometheus/Jaeger/OpenTelemetry Collector) descriptors within the root deploy/k8s directory for each Kubernetes variant platform.
In the rest-fights project, the generate-k8s-resources.sh script additionally copies in generated resources from the rest-heroes, rest-villains, rest-narration, and grpc-locations projects into the all-downstream.yml files in the deploy/k8s directory of the rest-fights project.
Docker compose resource generation follows a similar pattern as the Kubernetes resource generation.
Docker compose resources are generated into a deploy/docker-compose directory, either in the project root directory or in each individual project's directory.
Each Quarkus project (event-statistics, rest-fights, rest-heroes, rest-villains, rest-narration, grpc-locations) contains a src/main/docker-compose directory.
Inside this directory are a set of yaml files with a particular naming convention: infra.yml, java{{java-version}}.yml, and native.yml. Each of these files contains what we are calling Docker compose snippets. These snippets aren't a complete Docker compose file on their own. Instead, they contain service definitions that will ultimately end up inside the services block in a Docker compose file.
This table describes the different files that can be found inside a project's src/main/docker-compose directory.
| File name | Description |
|---|---|
infra.yml |
Any infrastructure definitions that are needed by the application. Definitions in here a re-used for each version of the application (i.e. JVM 17, Native). |
java{{java-version}}.yml |
Definition for the JVM version of application itself for a particular java version, denoted by {{java-version}}. |
native.yml |
Definition for the native image version of the application itself. |
The generate-docker-compose-resources.sh script loops through all versions of each application (Java version 17, both JVM and native - 8 total versions) and merges contents of these files from each project's src/main/docker-compose directory into each project's deploy/docker-compose directory as well as the respective files in the root deploy/docker-compose directory.
The generate-docker-compose-resources.sh script additionally creates the monitoring compose file (monitoring.yml) within the root deploy/docker-compose directory.