Use cache for steps

Im trying to move our tests from jenkins to circleci as POC.

In jenkins, I use containers steps which cache the requirements if they havent change.

my config.yml looks like this:

version: 2

    jobs:
      test:
        docker:
          - image: circleci/python:3.6.5-stretch
            environment:
              ENV: test
          - image: circleci/postgres:9.6-alpine
            environment:
              POSTGRES_USER: test
              POSTGRES_DB: test
        steps:
          - run:
              name: Waiting for Postgres to be ready
              command: |
                for i in `seq 1 10`;
                do
                  nc -z localhost 5432 && echo Success && exit 0
                  echo -n .
                  sleep 1
                done
                echo Failed waiting for Postgres && exit 1
          - run:
              name: installing OS requirements
              command: |
                sudo apt-get update
                sudo apt-get install unixodbc unixodbc-dev
                sudo pip install zope.interface constantly incremental attrs Automat twisted pymodbus3 pendulum redis influxdb msgpack-python toml python-etcd falcon waitress greenlet gevent logbook
          - checkout
          - run:
              name: install python requirements
              command: pipenv install --dev --skip-lock
          - run:
              name: run unittests
              command: export ENV=test && pipenv run pytest unittests/ -s

    workflows:
      version: 2
      unittests:
        jobs:
          - test

I would expect CircleCi to cache my requirements and only run the installations if that step is changes, and not on every run (which happens every push to github)

this is a major drawback for now because tests are takin X5 on circleci then they take on jenkins

EDIT: the problematic lines are the installing OS requirements and install python requirements runs

Jenkins setups tend to be dedicated, so the state of the OS is preserved between builds. Hosted CI tends to be wipe-clean and start from zero for each build. It’s slower, but it helps to guarantee your environment is the same for every build.

What I like to do is to:

  • Create another CircleCI build process for a build image
  • Base that build image on Alpine if possible
  • Put what I want on it (in your case Python and your other installations)
  • Push to a private/public Docker registry
  • Then use that as a parent image (in your case instead of circleci/python:3.6.5-stretch)
  • Your servers (Redis, Influxdb, Etcd) can probably all be pulled as secondary Docker images, rather than installed in the build server. They will all pull and start in parallel.