it’s always struck us as odd that downloading over the internet is faster than using a local cache
CircleCI builds aren’t running on a single bare metal machine: they run on a fleet of AWS machines. You may be thinking of the cache as a local disk, but the cache is stored in S3. S3, while in the same data center as the build machines, transfers slower than from DockerHub. Writing additional, separate caches will not slow down cache fetching.
I see from your other post that you’re using the machine executor. Please see my reply in that thread regarding docker layer caching.