Store_artifacts slow

Feedback & Bug Reports
,
1

Is there a way to speed up store_artifacts when uploading many files?
In my case it’s a code coverage report directory.
Originally our tests would take about 1.5 minutes to run. The upload of the code coverage report would take over 3 minutes. By stripping out everything not strictly necessary I’ve managed to reduce the upload to 2 minutes.
I’ve also tried to create a single unified HTML file using pandoc, but the result is pretty much garbage.
I’ve also tried to create a single ePub file using pandoc, but the result is just as garbage. The idea is that if you use the ePub add-on for Firefox, you’d still get a similar UX to viewing the HTML in the browser.

I’ve read these “solutions”:

  • Compress it first. This is not an acceptable solution as I want to be able to view the report easily from CircleCI
  • The background: true option is suggested but doesn’t exist (any more?).
2

CircleCI’s own recommendation is to compress all the files into a tar or zip file first, to reduce all the per-file processing that has to take place.

3

Yes, and as I have written above, I’ve read that, and concluded that it isn’t an acceptable solution.

4

That was the reason why I noted that it was CircleCI’s own recommendation. They have not published any other solution to the issue.

As for “background: true” link, as you may have found it through a general web search you may be unaware of what part of the CircleCI site that page is located in. “CircleCI Ideas” is an area to allow users to put forward ideas that CircleCi could include in future development work.

So currently “background: true” is an option that only exists on a run: command (basically an OS shell instance). Another limitation is that there is currently no published API for uploading an artifact, so it is not possible to emulate the store_artifacts command in such a way as to have it execute within a shell.

5

I’ve tried another option that theoretically should work. ReportGenerator can generate a single MHTML file, and while Chrome (but not Firefox) should support that, it doesn’t work when I open it. It only shows a blank page. Otherwise that would have been an acceptable solution.

6

For now I’ll leave it like this. It’s kind of absurd that uploading the test results takes more time than it takes to run the test. I think in the future I’ll add a mechanism to compress it, upload it to a server of ours that decompresses and serves it, and just put the link in the artifacts.

7

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.