Pre-signed URLs for downloading artifacts


#1

Scenario: I have several projects with interdependencies. When I build the first project, I want to trigger a build of the second project with the artifacts of the first project to make sure they work well together.

The problem: CircleCI stores artifacts, but there’s no especially great way to get them. It’s possible to create an API token for each project, but when there are many projects this is very tedious. Additionally, the 2nd project must maintain a list of all the API keys for all the projects which might trigger it, which is made even harder when developers want to build from forks. 10 projects * 25 developers = 250 keys I’d have to manage.

Proposed solution: for each build, auto-generate an API key which is valid for downloading the artifacts of that build. Or, allow a build to introspect its own API keys so it can choose to share it with the 2nd project. This is more or less equivalent with S3 pre-signed URLs, where the owner of the thing (the 1st project) decides that whatever receives some URL should be authorized to get that URL. The credentials get embedded in the URL, so the receiver can do an ordinary HTTP GET without needing to know any additional credentials. The URL is the credential.

Non-solution: rather than save the artifact to CircleCI, push it to the 2nd project with an HTTP post or similar.

The problem here is that the 2nd project is then required to maintain state of all the other projects that might trigger it, since it has no way to get those project’s artifacts after they have been pushed. Since CircleCI builds are stateless, this is pretty inconvenient.

It’s also potentially very slow. For large projects, collecting the artifacts can take minutes. Uploading them again to another project can take potentially minutes again.

Non-solution: generating a per-user API key which works for all projects.

The full capabilities of a user are significantly in excess of those required to download artifacts. This is a big principle of least authority violation.

The problem is further compounded since each project must maintain a copy of this key, so rotating it is very difficult since many projects must be altered. That also means every person in the organization knows this key. Consequently accountability is eliminated: if any employee wanted to do something malicious, they could use this key that everyone knows and avoid accountability. The key also needs to be rotated every time an employee leaves the organization, which is made very difficult given its duplication among all the projects.


#2