Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Smarter caching for big workspaces by default #95

@thomaseizinger

Description

@thomaseizinger

On large Rust repositories, it is easy to exceed GitHub's default cache limit of 10GB. I've recently been working on optimising our CI pipeline over at rust-libp2p. Here is what I landed on:

Our CI fans out testing into one job per crate. Applying this action naively completely busts the cache limit of Github because we would be saving and restoring 20+ caches per workflow run.

To combat this, I created a "cache factory" workflow that only runs on master and compiles the entire workspace: https://github.com/libp2p/rust-libp2p/blob/master/.github/workflows/cache-factory.yml. It is a bit more complicated in our case because we have different caches for the MSRV check and the actual testing but you get the overall idea.

More importantly, we still have other workflows (clippy, rustdoc, cross-compile to wasm etc) that always run on the entire workspace. Those use this action as is. That still creates a lot of caches, hence I am now adding

with:
  save-if: ${{ github.ref == 'refs/head/master' }}

to all cache invocations: libp2p/rust-libp2p#3249

Do you have any thoughts on that? I think that could actually be a safe default to add this action.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions