Convenience Images update: add our convenience image tooling to your own Docker images, with orbs

We’ve completed a set of orbs and orb commands designed to make the experience of using CircleCI convenience images available to folks using any CircleCI execution environment—from our machine or macos executors, to any third-party Docker images (including not just Debian/Ubuntu-based images but also those based on Alpine Linux).

What does this mean? Essentially, anyone who does not or cannot use CircleCI’s convenience images, for whatever reason, can now layer on the same tooling and functionality, at job runtime, using first-party CircleCI orbs.

Here is the list of orbs and orb commands that provide this functionality, in the rough order that one would likely want to run these commands in a CircleCI job:

build-tools-orb: install-ci-tools

(orb registry link, GitHub link)

This command installs a set of common packages that are often useful or required in CircleCI jobs—see source for the full list of packages.

jq-orb

(orb registry link, GitHub link)

This orb’s primary install command installs the jq JSON parser, which is required by many other first-party CircleCI orbs for doing things like parsing GitHub’s releases API to grab download links for a particular version of this or that tool.

docker-orb: install-docker-tools

(orb registry link, GitHub link)

This command installs the Docker CLI, as well as docker-compose and dockerize, two commonly used Docker tools.

node-orb: install

(orb registry link, GitHub link)

This command installs Node.js (any release can be installed), allowing users to essentially create a “Node variant” of any Docker image.

browser-tools-orb: install-browser-tools

(orb registry link, GitHub link)

This command installs Firefox, Geckodriver, Chrome (or, for Alpine Linux, Chromium), and ChromeDriver—e.g., the same functionality provided by a -browsers-tagged CircleCI convenience image.

Usage

Following is an example outlining the minimum required YAML configuration to run these commands:

version: 2.1

orbs:
  browser-tools: circleci/browser-tools@x.y.z
  build-tools: circleci/build-tools@x.y.z
  docker: circleci/docker@x.y.z
  jq: circleci/jq@x.y.z
  node: circleci/node@x.y.z

jobs:
  your-job:
    docker:
      - image: your-third/party-image:image-tag
    steps:
      - checkout
      - build-tools/install-ci-tools
      - jq/install
      - docker/install-docker-tools
      - browser-tools/install-browser-tools
      - node/install

workflows:
  your-workflow:
    jobs:
      - your-job

For all tools that support installation of specific versions, or installation to custom directory paths, these orb commands allow that level of customization, as well—so if your team requires a particular version of Docker, or Node.js, or Firefox, or jq (unlikely, but just in case), etc., you can now easily install the version you need, to the directory of your preference.

This work is part of a company-wide effort to make CircleCI work across as wide a range of execution environments as possible—and also parallels our work on a new set of convenience images designed to be faster, smaller, and more reliable than the current convenience images.

And, finally, all this code is open-source! So if you’d like to see an additional Docker-related tool, or an additional browser-testing tool, added to these orb commands—please file an issue or, even better, submit a pull request against our orb repositories :slightly_smiling_face:

3 Likes

build-tools/install-ci-tools errors out with openjdk12 images:

Sorry, you are using an unsupported OS/distribution (orb command is currently compatible across macOS, Alpine Linux, Debian Linux, and Ubuntu Linux)

This is in disagreement with the assertion here:

…we will be providing a way for folks to use any JDK 12 image and layer on the CircleCI customizations/tools…

CircleCI build here.

config yml available here.

Issue with primary openjdk 12 image was mitigated by using another image: adoptopenjdk/openjdk12:x86_64-ubuntu-jdk-12.0.1_12

1 Like

@manastalukdar do you happen to know the underlying Linux distribution of the image you were originally trying to use? That will help us investigate.

Please file this as an issue against https://github.com/circleci-public/build-tools-orb. Thank you!

We should also create a Java orb, so folks can just install OpenJDK at runtime… Then there won’t be this reliance on fickle/unreliable community-maintained upstream images :thinking:

1 Like

@rose, unfortunately, the openjdk docker hub page does not list the underlying Linux distro information. From the build-tools orb error message, I can only guess that it is not Alpine, Ubuntu or Debian, assuming the install-ci-tools step works as expected.

The image tags that I tried and did not work, are as follows: 12.0.1-jdk, 12.0.1, 12.0, 12, 12-jdk, 12-jdk-oraclelinux7. There were also other errors with some images and tags variations related to missing cryptographic keys, etc.

If circle-ci can provide a jdk12 base image that would be ideal.

@manastalukdar you can actually use another command in build-tools-orb to output this information:

https://circleci.com/orbs/registry/orb/circleci/build-tools#commands-env-info

env-info will gather information about the execution environment of whatever container it’s running in and output it to a file, which you can then store as an artifact. Can you give that a shot?

That said, I would guess that these images you’re trying are using Oracle Linux, as Oracle’s moving the OpenJDK 12 image from Ubuntu to Oracle Linux was the original cause of this entire debacle—and the reason we have been unable to provide an OpenJDK 12 image.

I will look into making our install-ci-tools command work with Oracle Linux (and also maybe output some basic environment info along with an error message).

Thanks!

2 Likes

@manastalukdar this issue has been patched in circleci/build-tools@2.4.0. Please see this pull request for details:

Briefly, the install-ci-tools command should now work with those OpenJDK 12 images from https://hub.docker.com/_/openjdk. Next up, we may have to make some of the same changes to our jq-orb, docker-orb, node-orb, and browser-tools-orb, to make sure their commands will also run on Oracle Linux. Appreciate your patience!

1 Like