Unarchiving cache occasionally takes over a minute instead of <5 sec

We’ve got test runs set up to execute with high parallelism and often, when I look to find the slowest container of the bunch, it turns out it took well over a minute to extract cached files in restore_cache that would otherwise get extracted in sub 5 seconds (and does indeed happen that fast on all the other parallel container runs).

This introduces a really painful bottleneck that I’m struggling to understand or do anything about…

Example logs from a randomly extremely slow restore_cache:

Found a cache from build 425006 at gem-cache-v8-........
Size: 177 MiB
Cached paths:
  * /home/circleci/project/vendor/bundle

Downloading cache archive...
Validating cache...
Download duration for cache file /tmp/cache3102993746: 3.245327422s

Unarchiving cache...
Extraction duration for gem-cache-v8-{{ arch }}-{{ checksum "Gemfile.lock" }}: 1m27.217016832s

I know that cache is tied to AWS and network issues can sometimes add latency, but as far as I can tell, this isn’t a case of that since the download happened in just a couple of seconds. There’s something really fishy going on, I’d really appreciate more insight here or suggestions to remedy the issue.