GHA pipeline rewrite for ease and speed

[gpmayorga]

Description

Main goal was to activate an efficient cache system to improve build time of the chain nodes and wasm builds. But since every single workflow file needed touching I started consolidating code and adding other improvements resulting in an almost full re-write of our pipelines.

Ref (internal) document: https://centrifuge.hackmd.io/Ftgot4ilSxK5ERyrEujnBg?view

Changes and Descriptions

Cache features

This PR introduces 3 types of cache systems:

1. Cross-PR and branch cache setup with a GCS bucket through "smart" Mozilla's sccache. Any workflow using Rust can take advantage of this by using actions/prep-ubuntu with some parameters
   Used by: sanity checks, benchmarks, linters
2. srtool WASM build cache
   Using the same cache system as our current pipelines with some enhancements (Swatinem/rust-cache). Getsmounted on the srtool container.
   Used by: wasm runtime builds
3. Docker layer caching with docker buildx
   Using buildx cache system, setup to use the registry as a cache but can be setup with GHA native cache system
   Used by: docker build

To review:

• Swatinem/rust-cache is only efficient if the cache name coincides. See https://github.com/centrifuge/centrifuge-chain/actions/caches. Problem is we only have 10GB or PRs will keep deleting each other caches (that's the issue right now), ideally main branch should build a cache that all PRs use.
• Sccache is now stored in Gcloud - This should hopefully keep the caches below 10G while having some shared cache for tests and checks (less critical than docker and wasm builds). Gcloud bucket is in the DEV account which means developers can potentially also use the cache from their laptops. https://console.cloud.google.com/storage/browser/centrifuge-chain-sccache-backend;tab=objects?forceOnBucketsSortingFiltering=true&project=peak-vista-185616&prefix=&forceOnObjectsSortingFiltering=false
• If we're way below the 10GB maybe we can keep Docker cache also in GHA cache system (faster), otherwise we'll keep it in the registry.

Benhcmarks

Now benchmarks will run on the main branch for every push and create a PR with the new weights (creating a PR from a bot has been disabled org-wide by an admin after this, woops!).
The benchmark job will upload the benchmark results to the job's page.
To review:

• Benchmark cache - The job takes over 1h each time, how to lower it?
• benchmark-test - Longest job on the "PR tests". Do we need it? Can we make it shorter/faster?

Docker builds

• Modified and improved Dockerfile for speed and stability, inspired in the polkadot-sdk Dockerfile.
• Standardize the Docker tags using automated docker-metadata.
  To review:
• Decide on the Docker metadata strategy
• If we have to tell a third party to take the "latest released tag". What would it be?
• Decide on the docker build cache: GHA backed.

New features/enhancements

• Code coverage: https://doc.rust-lang.org/rustc/instrument-coverage.html
• Build time graphs: https://doc.rust-lang.org/nightly/cargo/reference/timings.html
• (Optional) To be super rust-native, can we use https://github.com/matklad/cargo-xtask to define what needs "to be done" instead of using a bash script?

Checklist to set this PR as "ready"

• Try out sccache GHA module: does not work with srtool container (build wasms)
• Add a different cache system for WASM builds. -> Swatinem/rust-cache GH Action
• Add cache for Docker builds
• Review and refactor GHA workflows
  - [ ] Makefile to consolidate CI operations into a single source
• (Optional) Decide where to uplaod and store artifacts from the builds -> new storage bucket in prod account w/ (very) restricted access.
• Try out different machine sizes in GHA to find the most optimized $/time ratio.
• Play out with cache options and tag groups
• Testing caches:
  • WASM build
  • Docker build
  • Benchmarks
  • Tests and linters
• Figure out something clever to upload benchmarks regularly and ping devs (upload to the benchmark job)
• Try-runtime for runtime upgrades
• PR cleanup
• Review this PR with someone from Protocol (thanks @lemunozm and @wischli )
• Change the PR checks for the new ones

[wischli]

AFAICT, there are some removable artifacts left.

[wischli]

Thanks so much for taking the time to pimp up our CI. Looking awesome and blazingly fast 🚀

[lemunozm]

Why not development?

[wischli]

I thought we didnt need to check dev benchmarks to save time. I know this kind of contradicts my argument of keeping benchmarks in Dev. However, in the past releases I merged Centrifuge benchmarks to Dev because there was no runtime benchmark pipeline. I think thats feasible because this way we mimic mainnet.

How about we iterate over that when we build the next feature, which will probably first only exist in Dev runtime?

[gpmayorga]

Since this are PR checks, let's keep it minimal, this won't actually publish or set benchmarks anywhere, it's just to check that the benchmarks run
Do we really need all 3 to run?

[lemunozm]

👌🏻

[gpmayorga]

we could add @wischli too, for the bus factor

[lemunozm]

Out of curious: Why pointing to a hash directly?

[gpmayorga]

In the unlikely case that the actions/checkout repository introduces a bad update or let's say someone hacks it and introduces malicious code, our pipeline will run the update without questions if we do v1 vs the code commit. It's a way of freezing the code we run from others.
Keep in mind some of the CI pipelines have access to our Google Cloud credentials so a malicious change can potentially obtain access too.
More info: https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#using-third-party-actions

[lemunozm]

I think now Demo uses development we should add also here the weights. See this slack thread

[mustermeiszer]

I would also add development here. But lets have this in a second PR where we should also add the possibility to build development wasm for upgrades in dev and demo.

[lemunozm]

Should be list-taple reenable?

[wischli]

Our intention was to re-enable it in a follow up PR in case it doesnt work out of the box. WDYT?

[gpmayorga]

Every time I enabled it it failed consistently, I think it requires a bit of work, I agree on dealing with it in separate PRs

[lemunozm]

Can be this line removed?

[wischli]

This file is not used right now but kept as a wip state to introduce code cov.

[gpmayorga]

Yes, this will be a separate PR

[wischli]

Re-approving. Unfortunately, my approval is not sufficient.

[gpmayorga]

All tasks for the future:
https://centrifuge.hackmd.io/Ftgot4ilSxK5ERyrEujnBg?both#ToDo-after-merging-1551

[mustermeiszer]

I will approve based on the knoweledge I have but more on the fact that @wischli and @gpmayorga spent so much time on this. Thanks a lot for the improvements!!!

[mustermeiszer]

I would also add development here. But lets have this in a second PR where we should also add the possibility to build development wasm for upgrades in dev and demo.