Hey, folks - I’ve put together a build container that we use internally for our python builds.
We use Ubuntu 16.04 in production and were having issues with the differences between the Circle 1 environment, as well as drift between techniques we were using to make the environments match. And additionally, applying all the container customizations was adding minutes to the set up of every build.
Since Circle 2.0 can only use images from public registries, I created an image that gets all its secrets at runtime from environment variables or loading them from elsewhere (we use credstash).
The build is somewhat customized to our use case but should be easy to modify for anyone who would want to use it. Please note that this container is no way stable as we are currently changing to fit our needs as we migrate our builds to 2.0 - which is a polite of saying that if you want to use this, I’d suggest forking it.
Feedback, flames, pull requests welcome: email@example.com
The source for the image is here: https://github.com/KairosAerospace/dkr-circleci
Here’s the README:
A Docker container for use as a CircleCI 2.0 Primary Container
Available on Docker Hub as kairosaero/circleci-build.
This repository represents a primary container for a CircleCI 2.0
Docker Image Build
The image build does the following (turning the Dockerfile into a Docker image):
- Packages up an Ubuntu 16.04 Xenial userspace (the same as the Kairos production environment)
- Installs the production package loadout
- Installs Packer and Docker, since they are not pure
- Creates an empty Python 3.5 virtualenv and installs build prerequisites like
- Creates a directory structure expected by some Kairos software
- Installs a suite of build scripts into
/opt/kairos/binto run standard
build steps and puts them into
Docker Container Runtime
At container runtime, it expects the following environment variables to be defined:
CREDSTASH_TABLE- the credstash table to pull secrets from
SECRET_SET_NAME- the entry to pull from the credstash table
Given those variables, it then:
- Uses credstash to pull down JSON defining all environment variables
containing secrets (see
- Transforms the JSON with jq and injects the variable values into the
- Uses those credentials to wire the virtualenv to a private PyPI server
- Installs the Kairos build library from the private PyPI repo
- Writes a default config file for publishing packages to a private PyPI
- Activates the virtualenv for all docker commands run in the container
The CircleCI 2.0 Build Flow
An example, default
config.yml is included in this repository. It’s also included here:
version: 2 jobs: build: docker: - image: kairosaero/dkr-circleci entrypoint: . $(which kairos_env_init) && /bin/bash working_directory: /opt/kairos/build-home/repo steps: - checkout - run: name: environment setup command: kairos_env_init - run: name: python dependencies command: kairos_python_dependencies - run: name: python build command: kairos_python_build - run: name: python tests command: kairos_python_tests - run: name: python package command: kairos_python_package - run: name: python deploy command: kairos_python_deploy - run: name: python lambda publish command: kairos_lambda_publish - run: name: build cleanup command: kairos_python_cleanup - store_artifacts: path: test-output destination: test-output - store_artifacts: path: dist destination: build-artifacts - store_artifacts: path: logs destination: logs - store_test_results: path: test-output notify: webhooks: - url: https://bdoa77w3h0.execute-api.us-west-2.amazonaws.com/webhook/
Each of the various steps that are present in the Circle 1.0 builds is represented
and implemented in shell scripts called by the build. The inference is only applies
for Python builds and can be easily changed in the docker image or overridden in the
The steps do the following:
- The container starts with an empty virtualenv that only has packaging and setup tools installed.
environment setup- loads up all secrets and places them in the environment as well
as other environment tweaks.
python dependencies- installs from requirements.txt and writes out a verbose log to
logs/dependencies-install.log, which is turn preserved as an artifact.
python build- Plain, old
python setup.py install
python tests- runs
python setup.pywith appropriate options to log and run a coverage check.
These logs are also preserved.
python deploy- pushes packages out to a private PyPI repository, if that’s configured.
python lambda publish- special build step that will properly package and publish a pre-existing AWS Lambda.
build cleanup- preserves the virtualenv as an artifact to aid in troubleshooting.
Please note that the full “documentation” are the scripts themselves. They all live in
build_bin/ and are
/opt/kairos/bin in the Docker image.
This source code is made available under the MIT License. See LICENSE for more information.
© 2017 Kairos Aerospace. All Rights Reserved.