Securing deployment pipelines (CD setup with multiple team members)

Hey there!

I’m looking for tips how to secure our deployment pipeline, as I struggle to figure out the right structure in CircleCI that meets our requirements.

  • We are a small team of developers, and we are building a web application. The code is hosted on Github (in an org account).
  • Only a subset of specific developers has permission to access and administer our production infrastructure. (Let’s call them “admin developer” here.)
  • We want to automatically deploy our main branch to our production system as soon as a feature branch is merged, via a deployment pipeline on CircleCI.
  • We have a review process that requires approval from one other developer. The reviewer doesn’t have to be an admin developer, though.
  • The developers merge their branches on their own after review approval, so the act of merging isn’t (and shouldn’t have to be) done by an admin developer.
  • None of the regular (non-admin) developers should be able to have see the secret API keys that are needed to execute the production deployment.
  • Due to limitations at our hosting provider, we are unable to scope down the permissions of the API keys that we need for deploying. So as soon as someone knows these API keys, they can do whatever they want in our production infrastructure.

To sum it up: non-admin developers should be able to trigger code deployments autonomously via our automated deployment pipeline, but they shouldn’t be able to see the required (secret) API keys that facilitate this deployment.

As far as I understand, we cannot make this happen by storing the secrets in a CircleCI context, because we would then have to give every developer access to this context, and that way those people could theoretically obtain the secrets in the clear (e.g. by SSH’ing into the build machine).

Is there any way (or some other approach) that would allow us to achieve what we want? Or does someone maybe has any other advice on best practices for such a scenario?

Thanks!

The first thing to note is that it is just about impossible to ‘secure’ any build process if the process is expressed within the standard code base as everyone ends up with access to the config/script that details the process (config.yml for CircleCI) so they can review it or even modify it so that it shows any keys that are used by the script in any output stream.

The result is that everyone has to be creative and I think ends up with their own solution. For my environment, I have used the following as a way to put in place a level of control

  • Everything is expressed as parameter driven containers - we have one hell of a docker compose file that describes the platform and this file contains no hardcoded values that need to change across our target environments.

  • CircleCI publishes a list of IP addresses that can be used to impose controls on some third-party systems. The full details can be found here

It is quite a list and it can change, so my solution was to deploy self-hosted runners so that the CI process runs on systems I control with known IP addresses.

  • I use a third-party service for holding all my secrets and environment variables called Doppler. This allows access to be controlled by the IP address of the requesting system. As this is a third-party system the general access control is independent of the repo/CircleCI accounts.

  • Each target environment (dev[many], test[some], demo, QA, staging, production[a few]) has its own unique definition within Doppler with its own values and ACL.

The result is that only a job running on the self hosted runners is able to access secrets held at Doppler during a build and only each target environment only is able to access its environment variables.

  • Developers only develop and deploy to their development environments. All other targets just take known images from the development team and deploy them with different operation parameters that are also held in Doppler.

We are only a small company, so currently just using these tools means we are not going to pass any security audit, but the development team and ops team get to do just about anything they want/need on their target environments, but things become more restrictive as we progress along the release pipeline.

One issue is that such a workflow means that we do not operate as a single DevOps team. Operational tasks are separate from development.