Use cache for steps

Caching Dependencies
#1

Im trying to move our tests from jenkins to circleci as POC.

In jenkins, I use containers steps which cache the requirements if they havent change.

my config.yml looks like this:

version: 2

    jobs:
      test:
        docker:
          - image: circleci/python:3.6.5-stretch
            environment:
              ENV: test
          - image: circleci/postgres:9.6-alpine
            environment:
              POSTGRES_USER: test
              POSTGRES_DB: test
        steps:
          - run:
              name: Waiting for Postgres to be ready
              command: |
                for i in `seq 1 10`;
                do
                  nc -z localhost 5432 && echo Success && exit 0
                  echo -n .
                  sleep 1
                done
                echo Failed waiting for Postgres && exit 1
          - run:
              name: installing OS requirements
              command: |
                sudo apt-get update
                sudo apt-get install unixodbc unixodbc-dev
                sudo pip install zope.interface constantly incremental attrs Automat twisted pymodbus3 pendulum redis influxdb msgpack-python toml python-etcd falcon waitress greenlet gevent logbook
          - checkout
          - run:
              name: install python requirements
              command: pipenv install --dev --skip-lock
          - run:
              name: run unittests
              command: export ENV=test && pipenv run pytest unittests/ -s

    workflows:
      version: 2
      unittests:
        jobs:
          - test

I would expect CircleCi to cache my requirements and only run the installations if that step is changes, and not on every run (which happens every push to github)

this is a major drawback for now because tests are takin X5 on circleci then they take on jenkins

EDIT: the problematic lines are the installing OS requirements and install python requirements runs

#2

Jenkins setups tend to be dedicated, so the state of the OS is preserved between builds. Hosted CI tends to be wipe-clean and start from zero for each build. It’s slower, but it helps to guarantee your environment is the same for every build.

What I like to do is to:

  • Create another CircleCI build process for a build image
  • Base that build image on Alpine if possible
  • Put what I want on it (in your case Python and your other installations)
  • Push to a private/public Docker registry
  • Then use that as a parent image (in your case instead of circleci/python:3.6.5-stretch)
  • Your servers (Redis, Influxdb, Etcd) can probably all be pulled as secondary Docker images, rather than installed in the build server. They will all pull and start in parallel.