Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] Deduplicate actions/cache directives #340

Closed
3 tasks
joshlf opened this issue Sep 6, 2023 · 7 comments · Fixed by #835
Closed
3 tasks

[CI] Deduplicate actions/cache directives #340

joshlf opened this issue Sep 6, 2023 · 7 comments · Fixed by #835
Labels
compatibility-nonbreaking Changes that are (likely to be) non-breaking experience-easy This issue is easy, and shouldn't require much experience help wanted Extra attention is needed

Comments

@joshlf
Copy link
Member

joshlf commented Sep 6, 2023

We have a generate_cache job in CI which pre-populates a cache that is re-used by other jobs:

generate_cache:
runs-on: ubuntu-latest
name: Generate cache
steps:
- uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608 # v4.1.0
- uses: actions/cache@v3
with:
path: |
~/.cargo/
key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.toml') }}
- name: Populate cache
run: |
# Ensure all dependencies are downloaded - both for our crates and for
# tools we use in CI. We don't care about these tools succeeding for
# two reasons: First, this entire job is best-effort since it's just a
# performance optimization. Second, there may be failures due to
# issues other than failing to download dependencies (e.g., `cargo
# metadata` called with a malformed `Cargo.toml`, build failure in our
# own crate or in dependencies, etc). For those reasons, we discard
# stderr and ignore status codes.
#
# For downloading our crates' dependencies in particular, note that
# there is no support for doing this directly [1], so we just check
# all crates using --tests.
#
# [1] https://stackoverflow.com/a/42139535/836390
cargo check --workspace --tests &> /dev/null || true
cargo metadata &> /dev/null || true
cargo install cargo-readme --version 3.2.0 &> /dev/null || true
cargo install --locked kani-verifier &> /dev/null || true
cargo kani setup &> /dev/null || true

In each dependent job, we do the following:

  • Use needs: generate_cache
  • Use the following actions/cache action configuration:
    - uses: actions/cache@v3
    with:
    path: |
    ~/.cargo/
    key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.toml') }}

As of this writing, we copy that exact text five times. It'd be great if we could reduce the duplication.

Mentoring instructions

As proposed here by @mgeisler:

  • Create a separate cache.yml file in .github/workflows

  • Populate it with a job that performs these steps:

    - uses: actions/cache@v3
    with:
    path: |
    ~/.cargo/
    key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.toml') }}

  • Replace uses of that pattern in ci.yml with an invocation of that file, ie:

    - name: Populate cache
      uses: `.github/workflows/cache.yml`
Old/obsolete mentoring instructions

As discussed in this StackOverflow comment,

You can extract your own actions: docs.github.com/en/actions/creating-actions. YAML's own support for this isn't available in GHA though: github.com/actions/runner/issues/1182.

The task is to figure out whether there's a way to use that technique to define a new action in ci.yml that we can re-use in other steps. If there's an easy, simple syntax for this, we should do it. If it would require a lot of complexity, then it's not worth it.

@joshlf joshlf added help wanted Extra attention is needed experience-easy This issue is easy, and shouldn't require much experience compatibility-nonbreaking Changes that are (likely to be) non-breaking labels Sep 6, 2023
@joshlf joshlf added Hacktoberfest experience-medium This issue is of medium difficulty, and requires some experience and removed experience-easy This issue is easy, and shouldn't require much experience labels Sep 29, 2023
@zoo868e
Copy link
Contributor

zoo868e commented Oct 2, 2023

Hi @joshlf,
Is workflow_run what you're looking for?

@zoo868e
Copy link
Contributor

zoo868e commented Oct 2, 2023

I apologize for any inconvenience. It appears that the workflow_run isn't triggered when the Generate Cache Workflow is initiated by a pull request. Is it acceptable to employ the following workaround for this situation?

https://github.com/orgs/community/discussions/25220#discussioncomment-3246922

@joshlf
Copy link
Member Author

joshlf commented Oct 2, 2023

I apologize for any inconvenience. It appears that the workflow_run isn't triggered when the Generate Cache Workflow is initiated by a pull request. Is it acceptable to employ the following workaround for this situation?

https://github.com/orgs/community/discussions/25220#discussioncomment-3246922

Thanks for finding that workaround! Honestly probably not worth the complexity given that this is a very minor problem to begin with. If there's no good solution, it's probably fine to just leave it as-is. I'm gonna close this, but feel free to comment if you figure something else out!

@joshlf joshlf closed this as completed Oct 2, 2023
@mgeisler
Copy link

mgeisler commented Oct 4, 2023

Would it perhaps be interesting to use a local action? I do that in Comprehensive Rust:

This has worked out great for us in the last 12 months or so.

@joshlf
Copy link
Member Author

joshlf commented Oct 4, 2023

Oh yeah that's perfect! I'll re-open the issue with that suggestion. Thanks, @mgeisler!

@joshlf joshlf reopened this Oct 4, 2023
@joshlf joshlf added experience-easy This issue is easy, and shouldn't require much experience and removed experience-medium This issue is of medium difficulty, and requires some experience labels Oct 4, 2023
@Sh0g0-1758
Copy link
Contributor

@joshlf , I have created a PR as a patch for this issue. Can you please review it and suggest changes in it if required.

@dorryspears
Copy link
Contributor

@joshlf Hello, made a fix for this with #835. All tests pass
Happy weekend!

github-merge-queue bot pushed a commit that referenced this issue Feb 13, 2024
* fixed cache dupe call

* add shell

* Removed duplicate calls for caching in CI. Resolves #340

* fixed multiple runs

* removed whitespace .github/workflows/ci.yml

Co-authored-by: Joshua Liebow-Feeser <[email protected]>

* removed whitespace .github/workflows/ci.yml

Co-authored-by: Joshua Liebow-Feeser <[email protected]>

* removed whitespace .github/workflows/ci.yml

Co-authored-by: Joshua Liebow-Feeser <[email protected]>

---------

Co-authored-by: Joshua Liebow-Feeser <[email protected]>
dorryspears added a commit to dorryspears/zerocopy that referenced this issue Feb 20, 2024
* fixed cache dupe call

* add shell

* Removed duplicate calls for caching in CI. Resolves google#340

* fixed multiple runs

* removed whitespace .github/workflows/ci.yml

Co-authored-by: Joshua Liebow-Feeser <[email protected]>

* removed whitespace .github/workflows/ci.yml

Co-authored-by: Joshua Liebow-Feeser <[email protected]>

* removed whitespace .github/workflows/ci.yml

Co-authored-by: Joshua Liebow-Feeser <[email protected]>

---------

Co-authored-by: Joshua Liebow-Feeser <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compatibility-nonbreaking Changes that are (likely to be) non-breaking experience-easy This issue is easy, and shouldn't require much experience help wanted Extra attention is needed
Projects
None yet
5 participants