I figured out restoring the cache is very slow on MacOS.
I have created a test case with some node_modules for a React Native application. The following action is used:

uses: actions/cache@v2
with:
    path: 'node_modules'
    key: ${{ runner.os }}-node_modules-${{ hashFiles('**/yarn.lock') }}

Here are the results:

Restoring on ubuntu:
Test run 1 took 6s: https://github.com/andrmoel/github-actions-speed-test/runs/3476428716?check_suite_focus=true
Test run 2 took 8s: https://github.com/andrmoel/github-actions-speed-test/runs/3476523783?check_suite_focus=true

Restoring on macos:
Test run 1 took 3m25s: https://github.com/andrmoel/github-actions-speed-test/runs/3476430657?check_suite_focus=true
Test run 2 took 24s: https://github.com/andrmoel/github-actions-speed-test/runs/3476523785?check_suite_focus=true

In the best case restoring the cache on MacOS takes 3 times longer than on ubuntu. But what really makes me worry are the 3m25s from MacOS test run number 1.

Edit: I did some research on another repository and I assume the long action happens every time when the cache was hit the first time.

I have created a test repository with two identical github-actions. One action runs on ubuntu-latest the other one on macos-latest

https://github.com/andrmoel/github-actions-speed-test