Enable cross-OS caching for generic dirs. Enable OS-specific cache-compiled by default. #45

SaschaMann · 2023-01-09T11:56:33Z

Fixes #44

As pointed out in the upstream actions, symlinks may lead to breakage. I don't think that affects the cached directories, does it?

I moved the compiled caching to its own steps, that way we enable people to benefit from cross-OS caching where possible while still caching compiled.

Existing caches will no longer be restored due to the name change, so this may cause a wave of cache misses once released.

action.yml

SaschaMann · 2023-01-09T13:00:56Z

@giordano any idea what could be causing the failure? Does windows do something weird with the artifacts dir?

giordano · 2023-01-09T13:25:31Z

No, I don't think so 😕

rikhuijzer · 2023-01-09T13:38:59Z

action.yml

  cache-hit:
    description: 'A boolean value to indicate an exact match was found for the primary key. Returns \"\" when the key is new. Forwarded from actions/cache'
    value: ${{ steps.hit.outputs.cache-hit }}
+  cache-compiled-hit:


Maybe compiled-cache-hit?

I called it that at first but then decided to make it match the input cache-compiled. Happy to change it back though.

rikhuijzer · 2023-01-09T13:39:54Z

action.yml

        restore-keys: |
-          ${{ runner.os }}-test-${{ inputs.cache-name }}-
+          ${{ inputs.cache-name }}-
+        enableCrossOsArchive: true


The benefits would be mostly reduction of the required amount of storage or would there be more benefits?

Artifacts are mostly different per operating system (but they are indexed by a content hash, so they are safe to share across all operating systems), but the registry and the source of packages are the same, so keeping a single copy of them instead of 3 can save up some space in total (in particular the registry, which is typically the full git repository).

SaschaMann · 2023-01-09T15:55:10Z

I'll change the CI to test all combination of where a cache was created vs where it's being restored. Maybe that will make it clearer where the error comes from.

IanButterworth · 2023-07-26T20:07:08Z

Worth re-running CI? Logs have expired

giordano · 2023-07-27T02:28:51Z

There's a merge conflict now

fatteneder · 2023-10-29T14:23:16Z

There's a merge conflict now

I don't understand much about github ci but maybe this patch resolves the conflicts?

diff --cc action.yml
index 65ad5e6,d919963..0000000
--- a/action.yml
+++ b/action.yml
@@@ -34,25 -37,41 +37,43 @@@ runs
      - id: paths
        run: |
          [ "${{ inputs.cache-artifacts }}" = "true" ] && A_PATH="~/.julia/artifacts"
 -        echo "ARTIFACTS_PATH=$A_PATH" >> $GITHUB_ENV
 +        echo "artifacts-path=$A_PATH" >> $GITHUB_OUTPUT
          [ "${{ inputs.cache-packages }}" = "true" ] && P_PATH="~/.julia/packages"
 -        echo "PACKAGES_PATH=$P_PATH" >> $GITHUB_ENV
 +        echo "packages-path=$P_PATH" >> $GITHUB_OUTPUT
          [ "${{ inputs.cache-registries }}" = "true" ] && R_PATH="~/.julia/registries"
 -        echo "REGISTRIES_PATH=$R_PATH" >> $GITHUB_ENV
 +        echo "registries-path=$R_PATH" >> $GITHUB_OUTPUT
-         [ "${{ inputs.cache-compiled }}" = "true" ] && PCC_PATH="~/.julia/compiled"
-         echo "precompilation-cache-path=$PCC_PATH" >> $GITHUB_OUTPUT
        shell: bash
  
 -    - uses: actions/cache@58c146cc91c5b9e778e71775dfe9bf1442ad9a12
 +    - uses: actions/cache@704facf57e6136b1bc63b828d79edcd491f0ee84
        id: cache
        with:
-         path: "${{ format('{0}\n{1}\n{2}\n{3}', steps.paths.outputs.artifacts-path, steps.paths.outputs.packages-path, steps.paths.outputs.registries-path, steps.paths.outputs.precompilation-cache-path) }}"
-         key: ${{ runner.os }}-${{ inputs.cache-name }}-${{ hashFiles('**/Project.toml') }}
 -        path: "${{ format('{0}\n{1}\n{2}', env.ARTIFACTS_PATH, env.PACKAGES_PATH, env.REGISTRIES_PATH) }}"
++        path: "${{ format('{0}\n{1}\n{2}', steps.paths.outputs.artifacts-path, steps.paths.outputs.packages-path, steps.paths.outputs.registries-path) }}"
+         key: ${{ inputs.cache-name }}-${{ hashFiles('**/Project.toml') }}
          restore-keys: |
-           ${{ runner.os }}-${{ inputs.cache-name }}-
+           ${{ inputs.cache-name }}-
+         enableCrossOsArchive: true
  
      - id: hit
        run: echo "cache-hit=$CACHE_HIT" >> $GITHUB_OUTPUT
        env:
          CACHE_HIT: ${{ steps.cache.outputs.cache-hit }}
        shell: bash
+ 
++    # or should this be (to align with above)
++    # - uses: actions/cache@704facf57e6136b1bc63b828d79edcd491f0ee84
+     - uses: actions/cache@58c146cc91c5b9e778e71775dfe9bf1442ad9a12
+       id: cache-compiled
+       if: inputs.cache-compiled == 'true'
+       with:
+         path: ~/.julia/compiled
+         key: ${{ inputs.cache-name }}-compiled-${{ hashFiles('**/Project.toml') }}
+         restore-keys: |
+           ${{ inputs.cache-name }}-compiled-
+         enableCrossOsArchive: false
+ 
+     - id: compiled-hit
+       if: inputs.cache-compiled == 'true'
+       run: echo "cache-compiled-hit=$CACHE_HIT" >> $GITHUB_OUTPUT
+       env:
+         CACHE_HIT: ${{ steps.cache-compiled.outputs.cache-hit }}
+       shell: bash

IanButterworth · 2023-11-01T15:34:15Z

@SaschaMann let me know if you want help moving this forward (resolving the merge conflict)

IanButterworth · 2023-11-04T00:24:17Z

Hope you don't mind me going ahead. I want to make progress on JuliaLang/julia#50667

SaschaMann · 2023-11-04T13:48:33Z

@IanButterworth feel free to take over. I don't have the capacity to work on this atm unfortunately.

IanButterworth · 2023-11-06T15:34:12Z

action.yml

  cache-name:
    description: 'Name used as part of the cache keys'
-    default: 'julia-cache'
+    default: '${{ matrix.julia-version }}-${{ matrix.arch }}'


Note that I changed this, and it seems to work well.
Thankfully if a matrix variable doesn't exist it's interpolated as an empty string. So this should be pretty safe.

IanButterworth · 2023-11-06T15:48:30Z

I'm making the proposal to enable cache-compiled by default because:

it works well on 1.11
the cache-name default is now set up to be likely optimal for most use cases.
there is some benefit in julia <1.11 where caches for packages that don't depend on stdlibs can actually be used.
It would just be best to get this widely adopted (make ecosystem CI go brrr)

rikhuijzer · 2023-11-06T17:35:01Z

action.yml

        restore-keys: |
-          ${{ runner.os }}-${{ inputs.cache-name }}-
+          julia-cache-${{ inputs.cache-name }}-


Maybe we should remove the restore-keys altogether. The default examples at https://github.com/actions/cache don't seem to use it and I don't know exactly why it we decided to use it here

I think it's helpful for if there's been a slight change to the project that changes the hash. That shouldn't rule out using the cache I think.

(The main docs for using the action are in the GH docs, not the README, fwiw)

rikhuijzer

LGTM. I can't guarantee that merging this will not break workflows (for example, running out of space or taking longer on average), but at the same time caching is too hard to really figure out beforehand. I agree with Ian that getting the precompiled cache working is very important, so am in favor of moving forward.

SaschaMann · 2023-11-06T18:50:03Z

Happy for this to be merged if you all agree it's fine and I agree with Rik that it's hard to fully test this beforehand. However, if the CI failures are correct, they should be addressed. If not, maybe the CI needs fixing or removal.

Thanks for pushing this forward!

IanButterworth · 2023-11-14T04:29:38Z

Once JuliaLang/julia#52123 is merged my plan is to finish this up and get it released before 1.11 is out so we can make faster CI land with 1.11 nicely

IanButterworth · 2023-11-17T05:10:00Z

I've been testing now that JuliaLang/julia#52123 is merged and I'm seeing big speed ups in cached runs with no precompilation.

However, two thoughts:

I think it might not make sense to separate the caching of /compiled an other depot dirs, as they are all kind of interlinked.
Currently the compiled cache key isn't correct because it doesn't save the cache if any new precompilation has been done without changes to the project, which is possible if the registry has changed. Perhaps we should just always re-save it? If that's possible? Or is there something else that would be fast to hash that would be sensitive and specific-enough?

fatteneder · 2023-11-17T12:42:42Z

Or is there something else that would be fast to hash that would be sensitive and specific-enough?

What about Manifest.toml?
Is it possible to run a small Julia script in the inputs section, which resolves all Pkg versions.
That script could perhaps also generate a custom hash that accounts for arch, OS, Manifest, etc.

IanButterworth · 2023-11-17T12:50:20Z

That won't be sensitive to test dep version changes as the test env is a temp dir merge.

IanButterworth · 2023-11-17T14:23:23Z

I'm a bit stumped in trying to get hashFiles to hash the compiled dir.

One issue is that it will manually ignore any files outside of the github workspace, even if they're returned by glob.
However the 2nd arg of hashFiles currentWorkspace allows you to set a custom workspace during the check.

it doesn't expand the user and just does a string compare, so I submitted a PR to fix that hashFiles: expand user in currentWorkspace & show more in debug actions/toolkit#1582
Also it always adds a trailing path separator, which is a bit of a footgun

But I've set this up in a way I think should work and two issues:

The glob is returning all files not just .ji files
They are all being rejected for being outside of the current workspace.. (note that the log is misleading and makes it look like it is specifically controlled by GITHUB_WORKSPACE, but the currentWorkspace arg overrides this. Fixed in my PR)

##[debug]/home/runner/.julia/compiled/v1.11/Reexport
##[debug]Ignore '/home/runner/.julia/compiled/v1.11/Reexport' since it is not under GITHUB_WORKSPACE.
##[debug]/home/runner/.julia/compiled/v1.11/Reexport/bTpYr_AlfsU.ji
##[debug]Ignore '/home/runner/.julia/compiled/v1.11/Reexport/bTpYr_AlfsU.ji' since it is not under GITHUB_WORKSPACE.
##[debug]/home/runner/.julia/compiled/v1.11/Reexport/bTpYr_AlfsU.so
##[debug]Ignore '/home/runner/.julia/compiled/v1.11/Reexport/bTpYr_AlfsU.so' since it is not under GITHUB_WORKSPACE.
##[debug]/home/runner/.julia/compiled/v1.11/Reexport/bTpYr_sbGYn.ji
##[debug]Ignore '/home/runner/.julia/compiled/v1.11/Reexport/bTpYr_sbGYn.ji' since it is not under GITHUB_WORKSPACE.
##[debug]/home/runner/.julia/compiled/v1.11/Reexport/bTpYr_sbGYn.so
##[debug]Ignore '/home/runner/.julia/compiled/v1.11/Reexport/bTpYr_sbGYn.so' since it is not under GITHUB_WORKSPACE.

DilumAluthge · 2023-11-17T20:06:38Z

What's the benefit of cross-os caching? Precompile cachefiles (.ji and .so/.dylib/.jll files) are not portable across different operating systems.

IanButterworth · 2023-11-17T20:35:18Z

@DilumAluthge PR title was confusing. Updated

IanButterworth · 2023-11-17T20:38:34Z

Though I am starting to wonder whether it's just better to just maintain an OS-specific single cache of the depot.. so that everything is kept in lock-step

IanButterworth · 2023-11-18T13:52:46Z

After much debugging yesterday it turns out that the cache key is generated in the pre step, so we cannot make it depend on the state of the files being cached (unless actions/cache#673 is added).

That means we can't be smart about when to save a cache state, as we cannot rely on anything in the repo alone to inform that.

One strategy is to always save at the end, with the same cache key, but to do that we have to manually delete the old cache before saving it. There's no update mode in the API. It might require more user auth.

IanButterworth · 2023-11-18T15:12:02Z

I also think the enableCrossOsArchive: true thing won't work for us because basically always in CI the jobs are run concurrently, meaning the caches from different OS runs won't be able to sequentially coalesce content into the same cache file most of the time.

I'm thinking about opening another PR that:

just moves to create a single cache for all the stuff we care about in the depot (turns compiled on), as keeping everything in-sync is probably what we actually want to be able to avoid re-precompilation
adds caching of *_usage.toml logs so that auto Pkg.gc() can keep the depot trimmed down
has sensible matrix-informed defaults
always saves after a run to the same cache key (requires a manual deletion, detailed here)
adds retries to the cache download, incase the key was in the process of being replaced
Is a breaking change, which will help get people to think twice about whether they actually want a custom cache key, given the default should work best in most cases.

Though this model won't be ideal for cases where multiple jobs run concurrently on the same julia version & OS. That needs some more thought.

IanButterworth · 2023-11-19T05:21:09Z

Check out #71. I think it's a more sensible approach

IanButterworth · 2023-11-28T02:40:02Z

Now that #71 is merged I think this can be closed

SaschaMann requested a review from giordano January 9, 2023 11:56

SaschaMann marked this pull request as draft January 9, 2023 12:04

SaschaMann commented Jan 9, 2023

View reviewed changes

action.yml Outdated Show resolved Hide resolved

rikhuijzer reviewed Jan 9, 2023

View reviewed changes

SaschaMann mentioned this pull request Jan 9, 2023

Julia's version is not taken into account #46

Closed

IanButterworth mentioned this pull request Aug 28, 2023

Improve reuse of /compiled cache in CI JuliaLang/julia#50667

Closed

IanButterworth reviewed Nov 6, 2023

View reviewed changes

IanButterworth changed the title ~~Enable cross-OS caching~~ Enable cross-OS caching. Enable cache-compiled by default Nov 6, 2023

rikhuijzer reviewed Nov 6, 2023

View reviewed changes

rikhuijzer approved these changes Nov 6, 2023

View reviewed changes

SaschaMann marked this pull request as ready for review November 6, 2023 18:50

IanButterworth force-pushed the SaschaMann-patch-1 branch from 5a78f1f to 0464506 Compare November 17, 2023 20:27

IanButterworth changed the title ~~Enable cross-OS caching. Enable cache-compiled by default~~ Enable cross-OS caching for generic dirs. Enable OS-specific cache-compiled by default. Nov 17, 2023

Enable cross-OS caching

41e5de1

IanButterworth force-pushed the SaschaMann-patch-1 branch from b3cf521 to 41e5de1 Compare November 17, 2023 20:57

IanButterworth added 6 commits November 17, 2023 20:29

debug

84de0b3

tryfix

8c39332

retry

f5ce23f

debug

bc86dea

debug

a47e118

try updated cache

8608e36

IanButterworth mentioned this pull request Nov 19, 2023

Update cache every run. Add /compiled and /logs. Make key sensitive to matrix. #71

Merged

IanButterworth closed this Nov 28, 2023

IanButterworth deleted the SaschaMann-patch-1 branch November 28, 2023 02:40

Enable cross-OS caching for generic dirs. Enable OS-specific cache-compiled by default. #45

Enable cross-OS caching for generic dirs. Enable OS-specific cache-compiled by default. #45

Uh oh!

Conversation

SaschaMann commented Jan 9, 2023

Uh oh!

Uh oh!

SaschaMann commented Jan 9, 2023

Uh oh!

giordano commented Jan 9, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SaschaMann commented Jan 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

IanButterworth commented Jul 26, 2023

Uh oh!

giordano commented Jul 27, 2023

Uh oh!

fatteneder commented Oct 29, 2023

Uh oh!

IanButterworth commented Nov 1, 2023

Uh oh!

IanButterworth commented Nov 4, 2023

Uh oh!

SaschaMann commented Nov 4, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

IanButterworth commented Nov 6, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rikhuijzer left a comment

Choose a reason for hiding this comment

Uh oh!

SaschaMann commented Nov 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

IanButterworth commented Nov 14, 2023

Uh oh!

IanButterworth commented Nov 17, 2023

Uh oh!

fatteneder commented Nov 17, 2023

Uh oh!

IanButterworth commented Nov 17, 2023

Uh oh!

IanButterworth commented Nov 17, 2023

Uh oh!

DilumAluthge commented Nov 17, 2023

Uh oh!

IanButterworth commented Nov 17, 2023

Uh oh!

IanButterworth commented Nov 17, 2023

Uh oh!

IanButterworth commented Nov 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

IanButterworth commented Nov 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

IanButterworth commented Nov 19, 2023

Uh oh!

IanButterworth commented Nov 28, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

SaschaMann commented Jan 9, 2023 •

edited

Loading

SaschaMann commented Nov 6, 2023 •

edited

Loading

IanButterworth commented Nov 18, 2023 •

edited

Loading

IanButterworth commented Nov 18, 2023 •

edited

Loading