Example of python builds on Ubuntu 16 Xenial for Circle 2.0

docker
container-image
python

#1

Hey, folks - I’ve put together a build container that we use internally for our python builds.

We use Ubuntu 16.04 in production and were having issues with the differences between the Circle 1 environment, as well as drift between techniques we were using to make the environments match. And additionally, applying all the container customizations was adding minutes to the set up of every build.

Since Circle 2.0 can only use images from public registries, I created an image that gets all its secrets at runtime from environment variables or loading them from elsewhere (we use credstash).

The build is somewhat customized to our use case but should be easy to modify for anyone who would want to use it. Please note that this container is no way stable as we are currently changing to fit our needs as we migrate our builds to 2.0 - which is a polite of saying that if you want to use this, I’d suggest forking it.

Feedback, flames, pull requests welcome: ari@kairosaerospace.com

See: https://hub.docker.com/r/kairosaero/dkr-circleci/

The source for the image is here: https://github.com/KairosAerospace/dkr-circleci

Here’s the README:

kairosaero/circleci-build

A Docker container for use as a CircleCI 2.0 Primary Container

Available on Docker Hub as kairosaero/circleci-build.

Overview

This repository represents a primary container for a CircleCI 2.0
containerized build.

Docker Image Build

The image build does the following (turning the Dockerfile into a Docker image):

  1. Packages up an Ubuntu 16.04 Xenial userspace (the same as the Kairos production environment)
  2. Installs the production package loadout
  3. Installs Packer and Docker, since they are not pure apt installs
  4. Creates an empty Python 3.5 virtualenv and installs build prerequisites like
    twine, setuptools, and credstash
  5. Creates a directory structure expected by some Kairos software
    (/opt/kairos/*)
  6. Installs a suite of build scripts into /opt/kairos/bin to run standard
    build steps and puts them into $PATH.

Docker Container Runtime

At container runtime, it expects the following environment variables to be defined:

  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY
  • AWS_DEFAULT_REGION

Optionally:

  • CREDSTASH_TABLE - the credstash table to pull secrets from
    (default: circleci-secrets)
  • SECRET_SET_NAME - the entry to pull from the credstash table
    (default: default-build-secrets)

Given those variables, it then:

  1. Uses credstash to pull down JSON defining all environment variables
    containing secrets (see build_secrets_template.json)
  2. Transforms the JSON with jq and injects the variable values into the
    environment
  3. Uses those credentials to wire the virtualenv to a private PyPI server
  4. Installs the Kairos build library from the private PyPI repo
  5. Writes a default config file for publishing packages to a private PyPI
    server
  6. Activates the virtualenv for all docker commands run in the container

The CircleCI 2.0 Build Flow

An example, default config.yml is included in this repository. It’s also included here:

version: 2
jobs:
  build:
    docker:
      - image: kairosaero/dkr-circleci
        entrypoint: . $(which kairos_env_init) && /bin/bash
    working_directory: /opt/kairos/build-home/repo
    steps:
      - checkout
      - run:
         name: environment setup
         command: kairos_env_init
      - run:
          name: python dependencies
          command: kairos_python_dependencies
      - run:
          name: python build
          command: kairos_python_build
      - run:
          name: python tests
          command: kairos_python_tests
      - run:
          name: python package
          command: kairos_python_package
      - run:
          name: python deploy
          command: kairos_python_deploy
      - run:
          name: python lambda publish
          command: kairos_lambda_publish
      - run:
          name: build cleanup
          command: kairos_python_cleanup
      - store_artifacts:
          path: test-output
          destination: test-output
      - store_artifacts:
          path: dist
          destination: build-artifacts
      - store_artifacts:
          path: logs
          destination: logs
      - store_test_results:
          path: test-output
notify:
  webhooks:
    - url: https://bdoa77w3h0.execute-api.us-west-2.amazonaws.com/webhook/

Each of the various steps that are present in the Circle 1.0 builds is represented
and implemented in shell scripts called by the build. The inference is only applies
for Python builds and can be easily changed in the docker image or overridden in the config.yml.

The steps do the following:

  1. The container starts with an empty virtualenv that only has packaging and setup tools installed.
  2. environment setup - loads up all secrets and places them in the environment as well
    as other environment tweaks.
  3. python dependencies - installs from requirements.txt and writes out a verbose log to
    logs/dependencies-install.log, which is turn preserved as an artifact.
  4. python build - Plain, old python setup.py install
  5. python tests - runs python setup.py with appropriate options to log and run a coverage check.
    These logs are also preserved.
  6. python deploy - pushes packages out to a private PyPI repository, if that’s configured.
  7. python lambda publish - special build step that will properly package and publish a pre-existing AWS Lambda.
  8. build cleanup - preserves the virtualenv as an artifact to aid in troubleshooting.

Please note that the full “documentation” are the scripts themselves. They all live in build_bin/ and are
installed into /opt/kairos/bin in the Docker image.

License

This source code is made available under the MIT License. See LICENSE for more information.

© 2017 Kairos Aerospace. All Rights Reserved.


#2

Thank you for writing this detailed example.

This will be useful for many of our customers. :thumbsup:


#3

This is awesome. Thank you for sharing :slight_smile:


#4

This topic was automatically closed 41 days after the last reply. New replies are no longer allowed.