When CircleCI is told to upload a directory artifact, it is uploaded one file at a time, which ends up being extremely slow. Here is a build with two modes of artifact uploading (with this configuration):
- various intermediate build steps that we generally don’t need to check often, so it’s uploaded as a tarball; and,
- documentation that we generally do want to check (if documentation was changed in the PR), so it’s preferable to upload individually for easy online browsing and it’s uploaded as a directory, recursively.
The intermediate steps consist of 10693 files in 107 directories totalling 90M. The compressed tarball produced by CircleCI is 53.7 MB. From the build log, it took 0s to produce the tarball and 1s to upload it as an artifact.
On a local build (I did not want to try to download everything individually), the documentation consists of 7285 files in 112 directories totalling 132M. From the build log, it took 5m53s to upload.
In summary, the recursive upload was of 2.45 the size as the tarball, but it took 353 times longer to upload. That is extremely slow. I would guess it’s creating a new connection for each file, or trying to record in an artifact database the link to each file. It would be so much better for build times (both ours and just for general efficiency’s sake) if this could be batched somehow, e.g., tar|upload|untar or similar.