Next-Gen CircleCI Convenience Images - Public Beta

Ecosystem CircleCI Images
, , ,
1

Hey Circlers (just completely made that up),

For those who don’t know, we at CircleCI maintain a fleet of Docker images, which we call Convenience Images, for various languages and databases. These images are designed to run well in a CI environment as opposed to production or local environments. They exist to serve as a quick and convenient starting point for users.

The Problem with the Current/Legacy Images

The Docker build system we are using is old and complicated so we’ve been designing a new suite of images. Our 14 images are based off of 14, separate, upstream Docker Library images which weren’t designed for CI. This has caused surprise breaking changes for us and users, bloated images, and an inefficient build system.

New/Next-Gen Images

Our new images are being built from the ground up with CI, efficiency, and determinism in mind. Here’s some of the “highlights”:

  • Faster “Spin-Up” time
    • In Docker terminology, the new images tend to have less layers and each layer tends to be smaller. This is due to installing less bloat, only the tools most of our users need, as well as following Dockerfile best practices to make images efficient. In short, using these new images will lead to faster image downloads when a build starts, and a higher likelihood that the image is already cached on the host.
  • Improved reliability & stability
    • Currently images are rebuilt practically everyday, which potential changes from upstream that we can’t always test fast enough. This isn’t good for stable builds that prevent breaking changes. Next-gen images will only be rebuilt for security and critical-bugs, leading to more stable, more deterministic images.
  • Automated Convenience Image Documentation
    • This is still a work in progress. The Next-Gen image build system is being designed to make automated documentation for images a possibility. It will be much, much, easier in the future to understand how Convenience Images work, what software they contain, which tags are available, and more. Stay tuned.

Public Beta Images

We have a few Next-Gen images that have reached “public beta status” with more on the way in the next couple of months. You’re welcome to start using these images (below) and provide your feedback. The feedback we’re looking for:

  1. Feedback on if this image is working for your project or if it’s missing something or doing something “wrong” in your opinion. Particularly for a “language image”, is this image doing something that goes against community-recommendations for that specific programming language.
  2. Feedback on how this image is working for you if you’re coming from an equivalent legacy image.
  3. Good feedback. Is this working perfectly for you? Let us know that too.
  4. Anything else you think you should add.

Feedback can be posted here on this topic, the public beta CircleCI Discuss topic for a specific image, or in a GitHub Issue for a specific image.

Public Beta Images:

Name Image GitHub Discuss Post Notes
Clojure cimg/clojure link link The Clojure programming language.

Looking for a missing image? cimg/base, cimg/go, cimg/openjdk, cimg/node, cimg/php, and cimg/rust have all been promoted to GA/stable.

We have 4 more images currently in alpha status. Check back here regularly for updates on the public beta program. There’s a button at the bottom of this topic where you can “watch” or “track” this topic.

Everything is open source and designed for you. Contributions welcome. Any questions, please let us know, thanks!

4 Likes
2

The Base and Go images have now been promoted to GA. They are stable and ready to go for everyone!

3

A post was split to a new topic: Ruby image started failing for me

4

If I want to propose a new image type, is that possible? I currently have built something of cimg/base and want to know if there is a possibility for it to become an officialy convenience image

tool: packer

5

We don’t have a program to accept 3rd party images at this time but it’s something we’re looking into for later this year.

If you want to create your own CircleCI-ready image though then you’re on the right track, I’d base it off of cimg/base just like you said. That’s the best way to do it.

6

Can you explain why this would be true? Won’t there still be a large number of different images?

Spin up time is often a very big part of a job’s execution time, so anything in this area is interesting.

7

When a CircleCI build host starts a job with the Docker executor, the Docker image is downloaded. Basically a docker pull. This image then remains cached on the host for a set time. The next-gen Convenience Images are designed to inherit layers from other images whenever possible and for there to be less changes. To generalize a bit, what happens is we can go from thousands of individual images to just a few hundred. This great increases the chance that an image may already be on the host, meaning we can skip the download, or at least most of the layers will be on the host, meaning less to download.

8

Q1) What r your plans for Windows dockerfiles? I have been using Docker on Windows and they seem to work very well - both Linux and Window images.
Q2) I am interested in your cimg/python. How do we get the right version of Python, and what does a user do to add additional modules, eg pytest etc.

You may already have answers to both questions. If so, url references would be helpful.

Thanks

9

Hi, great questions.

  1. There’s no immediate plans but we’re open to feedback. Considering CircleCI has Windows machine/VM images, when would you want to use Docker images on Windows rather then just use the Docker executor or Windows executor?
  2. The Docker tag is the version for the image. So for example, if you want to use Python v3.7.7, you use the fulle image: cimg/python:3.7.7. To install modules, you can do the same thing you might do on your local machine. You can install them via apt-get and you can install them via pip.
10

Thanks for your fast response and keeping an open mind. I’ll try to answer from my point of view and hope more evidence to support it
comes from other users, etc.

  1. There’s no immediate plans but we’re open to feedback. …

The technical advantages of Windows Dockerfile over Windows VM are the same as Linux Dockerfile over Linux VM. Docker machines load faster, there are more variety available in Docker Hub etc.

Also, Microsoft does not charge for running the Dockerfiles, and Windows 10/Server can run both Linux and Window dockerfiles simultaneously.

As an example, circleci/windows@2.2.0 has VS 2019 installed even though there is no way to run to GUI IDE. VS2019 Community is free yes, but the installation takes a lot of space not typically needed for a build/test machine. The less bulky way is to just use VS2019 build tools or Windows/Linux docker mcr.microsft.com/dotnet/core.

The business reason for supporting Windows dockerfiles is to attract more Windows customers to use CircleCI. Many software companies support both Linux and Windows, so allowing them to option of hosting Docker in Linux or Windows enable them to maximize their initial machine investment, even as we :slight_smile: cleverly persuade them to use Linux in the long run.

Finally, by using Docker for both Windows and Linux, the platform differences are much reduced. For eg, I use bash on Windows simply because I prefer one CircleCI config to handle both platforms. That way, I dont have to use Powershell on Windows, Bash on Linux.

In the coming months, Windows 10 will have WSL ( Windows Subsystem for Linux ), so that the same CircleCI config can probably run both Windows and Linux, just by sticking to the linux subset that works for Windows. CircleCI would probaby go with Windows dockers with WSL enabled for the convenience of everyone.

  1. The Docker tag is the version for the image. …

Yes, I found the workaround. However, I had thought that should no version be given, the intention is latest-version, and I believe CircleCI images can be set up that way. In other words, cimg/python -> cimg/python:latest -> cimg/python:3.8

Have a nice day!

11

I’m getting a lot of notices to move to next gen images and that the legacy images will be deprecated, but there’s still no MySQL image listed. Are you not planning to provide one?

2 Likes