Share system level dependencies between workflow steps

Build Environment Caching Dependencies
1

I have several steps in my workflow and all of them require system level dependencies, like ruby, python etc…
I created a reference that deal with it and reusing it in every step. It works, but it still executes the installation every step.
Here is the reference:

        version: 2
    references:
      container_config: &container_config
        ...

      system_dependencies: &system_dependencies
        run:
          name: Install system requirements
          command: |
            sudo apt-get install python-pip python-dev ruby-full
            sudo pip install mozdownload mozinstall
            ...

And I am currently using it like so:

deploy-staging:
<<: *container_config
steps:
  - *system_dependencies

Is there a way to reuse already installed packages or cache all of them? Or any other approach…

2

Yes, assuming you’re using the Docker executor. Create a separate build process to generate a Docker image and push it to a Docker image registry. This can inherit from your existing CircleCI convenience image and then add the new things you want.

You can then specify a custom image in your build process, and it will use that rather than the one you’re using now.

3

It is A way, but that us not what I meant. I prefer to use standard images.

4

It’s the best way, since it is in keeping with how Docker works. Docker images are designed to be extended.

The only other alternative I can think of is caching the binary files that would be installed using apt-get (e.g. in /usr/bin) but that is not sufficient, since you’ll need to cache the changes made to /lib, /etc and /var. That is going to get very messy, very quickly.

5

Again, that was not my question…
Thanks for the advice though

6

It actually was your question. If you have constraints you wish to apply when addressing volunteers, it is a kindness to state them up-front.

Why so? If you want the most fitting advice here, it may help to explain this requirement.

7

If you’re fine to not use the Debian/Ubuntu supplied packages when you should be able to achieve this with some manual installs using a custom install location, a sprinkle of path manipulation and using CircleCI’s workspaces. Though this may come with some caveats.

I noticed that you’re installing python-pip etc, have you checked out the other pre-built images provided by CircleCI. The python ones include pyenv for the different versions, so you can easily add another version of python there and pip is already installed.

Also, might I suggest that you use pipsi to install the CLI like tools. It’s a great little tool to wrap entry point console scripts like this for use on the CLI making them “globally” available to the user across virtualenvs without the need to do a sudo install. While not essential in a CI environment, the virtualenv that it creates can also be moved between workflow steps in a workspace very easily.

2 Likes
8

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.