CircleCI CLI: Error response from daemon: invalid UTS mode

Build Environment
, ,
1

I"m trying to run the “demo” container in my local Ubuntu box as instructed here (sorry can’t use links):

docs/how-to-use-the-circleci-local-cli/#run-a-job-in-a-container-on-your-machine

As soon as I execute the following command circleci local execute build I get an error when the postgres container is being created:

Fetching latest build environment...
Docker image digest: sha256:15e0f186167e1678cc158263ccf805cef74af3ab6c3273703b3fde0fe0ad5c50
====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 23.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: systemd
 Cgroup Version: 2
 Kernel Version: 5.19.0-41-generic
 Operating System: Ubuntu 22.04.2 LTS
 OSType: linux
 Architecture: x86_64

Starting container circleci/golang:1.12
Warning: No authentication provided, using CircleCI credentials for pulls from Docker Hub.
  image is cached as circleci/golang:1.12, but refreshing...
circleci/golang:1.12:
  using image circleci/golang@sha256:ae191834590b2cdee6ca9bb6985f02e05b3f9b326536f83494f788889481b408
  pull stats: Image was already available so the image was not pulled
  time to create container: 27ms
Starting container circleci/postgres:12.1-alpine
1.12: Pulling from circleci/golang
Digest: sha256:ae191834590b2cdee6ca9bb6985f02e05b3f9b326536f83494f788889481b408
Status: Image is up to date for circleci/golang:1.12
Creating Docker containers in parallel ....Error:   error starting container circleci/postgres:12.1-alpine: Error response from daemon: invalid UTS mode: containerf7fff5f72e9a17d29dfa6084ca4d2013c4578d71f5607715b83edce17d695687

Error: 
Unexpected environment preparation error: Error response from daemon: invalid UTS mode: containerf7fff5f72e9a17d29dfa6084ca4d2013c4578d71f5607715b83edce17d695687

Step failed

There seems to be a post in StackOverflow from another person experiencing the same but not many responses. Wondering if this could be something specific to Ubuntu?

Running the hello world and other docker images from the hub work just fine. Only CircleCI images seem to have this problem. I’m trying to debug something locally for our build environment custom image but of course I can’t even get past the part of running the “demo” one from CircleCI. Trying to run steps from our build setup raises the same error.

Thanks in advance!

2

The first thing to look at will be the postgress image you are currently using as

circleci/postgres:12.1-alpine

Is a superseded legacy image from about 3 years ago. Current images can be found at the following link

It is not clear from an internet search what an invalid UTS error is, but over the 3 years, a lot has happened to the Docker environment.

If that does not help can you upload your config.yml file.

3

Interesting, that is the image that the demo from the public circleci docs points to, so I guess the documentation is not up to date? Can’t post links I guess for not having enough posts but if you google “how-to-use-the-circleci-local-cli” it’ll take you to the page I’m talking about.

I did go to the github repo from CircleCI and tried to run one of their ruby on rails images by checking out the demo repo, it builds but then fails with some ruby version mismatch error.

In any case, this is our current .circleci/config.yml file (some sensitive info removed) :

version: 2.1 # Use 2.1 to enable using orbs and other features.

# Declare the orbs that we'll use in our config.
# read more about orbs: https://circleci.com/docs/2.0/orb-intro/
orbs:
  ruby: circleci/ruby@1.7.1
  node: circleci/node@5.1.0
  browser-tools: circleci/browser-tools@1.4.1

jobs:
  build: # our first job, named "build"
    # we run "parallel job containers" to enable speeding up our tests;
    # this splits our tests across multiple containers.
    parallelism: 6

    docker:
      - image: cimg/ruby:2.7.5-browsers # this is our primary docker image, where step commands run.
        # auth:
        #   username: mydockerhub-user
        #   password: $DOCKERHUB_PASSWORD  # context / project UI env-var reference
      - image: cimg/postgres:13.5
        # auth:
        #   username: mydockerhub-user
        #   password: $DOCKERHUB_PASSWORD  # context / project UI env-var reference
        environment: # add POSTGRES environment variables.
          POSTGRES_USER: circleci-demo-ruby
          POSTGRES_DB: app_testing
          POSTGRES_PASSWORD: ""
    # environment variables specific to Ruby/Rails, applied to the primary container.
    environment:
      BUNDLE_JOBS: "3"
      BUNDLE_RETRY: "3"
      PGHOST: 127.0.0.1
      PGUSER: circleci-demo-ruby
      PGPASSWORD: ""
      RAILS_ENV: test
      RUBYOPT: "-W:deprecated"
    steps:
      # Add SSH key for access to repo
      - add_ssh_keys:
          fingerprints:
            - "removed"
      - checkout # pull down our git code.
      - ruby/install-deps # use the ruby orb to install dependencies

      # Install chrome things for capybara
      - browser-tools/install-chrome
      - browser-tools/install-chromedriver
      - run:
          command: |
            google-chrome --version
            chromedriver --version
          name: Check install

      - node/install:
          install-yarn: true
      - run: node --version

      # use the node orb to install our packages
      # specifying that we use `yarn` and to cache dependencies with `yarn.lock`
      # learn more: https://circleci.com/docs/2.0/caching/
      - node/install-packages:
          pkg-manager: yarn
      - run:
          name: Wait for DB
          command: dockerize -wait tcp://localhost:5432 -timeout 1m
      - run:
          name: Database setup
          command: bundle exec rails db:schema:load --trace

      - run:
          name: Asset Precompilation
          command: bundle exec rails assets:precompile

      # Run rspec in parallel (if enabled with more than one container in the "parallelism" setting at the top of the config)
      - ruby/rspec-test:
          order: rand

# We use workflows to orchestrate the jobs that we declared above.
workflows:
  version: 2
  build_and_test:     # The name of our workflow is "build_and_test"
    jobs:             # The list of jobs we run as part of this workflow.
      - build         # Run build.
      # There used to be another "test" job after this but it seemed superfluous and I was
      # able to get all the tests to run just in the "build" job which seems to have saved
      # an extra minute of setup time.

As per the documentation I mentioned above, it states the following:

If your CircleCI configuration is set to version 2.1 or greater, you must first export your configuration to process.yml , and specify it when executing with the following commands:

circleci config process .circleci/config.yml > process.yml
circleci local execute -c process.yml JOB_NAME

Which I did to create the mentioned process.yml file. Then, I would try to run the circleci local execute part with the generated process file and the job name but I still get the same UTS error.

I tried changing the postgres image from 13.2 (which is what we use in the real world) to the latest version cimg/postgres:15.2 but it still fails with the same error:

Fetching latest build environment...
Docker image digest: sha256:b27e837e0acb509680f924ed3099e1917f3f6574c23617c4dfbb7a7c8ffadbf9
====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 23.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: systemd
 Cgroup Version: 2
 Kernel Version: 5.19.0-41-generic
 Operating System: Ubuntu 22.04.2 LTS
 OSType: linux
 Architecture: x86_64

Starting container cimg/ruby:2.7.5-browsers
Warning: No authentication provided, using CircleCI credentials for pulls from Docker Hub.
  image is cached as cimg/ruby:2.7.5-browsers, but refreshing...
cimg/ruby:2.7.5-browsers:
  using image cimg/ruby@sha256:08391a7e5c03a4ac22234eee0487a312eb1991d1e70e228696a99a95dbd3cfda
  pull stats: Image was already available so the image was not pulled
  time to create container: 27ms
Starting container cimg/postgres:15.2
2.7.5-browsers: Pulling from cimg/ruby
Digest: sha256:08391a7e5c03a4ac22234eee0487a312eb1991d1e70e228696a99a95dbd3cfda
Status: Image is up to date for cimg/ruby:2.7.5-browsers
Creating Docker containers in parallel ..........................Error:   error starting container cimg/postgres:15.2: Error response from daemon: invalid UTS mode: container559c90d7de5e95b96d958ce6e06949695fc38c0d0c0610bda5234f23fca858c8

Error: 
Unexpected environment preparation error: Error response from daemon: invalid UTS mode: container559c90d7de5e95b96d958ce6e06949695fc38c0d0c0610bda5234f23fca858c8

Step failed

Error: Unhandled prepare executor error: Error response from daemon: invalid UTS mode: container559c90d7de5e95b96d958ce6e06949695fc38c0d0c0610bda5234f23fca858c8

I’m on Linux so am not sure if this is a problem with docker/ubuntu itself, will ask tomorrow someone from my team to test under macOS. Apppreciate your help!

4

OK, you may have to raise a support ticket, but you can pass this thread on as part of the ticket as that may speed things up.

I’ve created a basic project with your config.yml file and while I can not test at the yarn level as that comes from your project, it is not required.

The issue you are seeing is coming right at the start of a run in the ‘Spin up environment’ section. Using your config.yml I see no errors and my CI job continues to execute the following sections without issues. Below is what I see reported in the ‘Spin up environment’ section, which is very different from what you see,

Build-agent version 1.0.173685-09e116e6 (2023-05-17T16:52:57+0000)
System information:
 Server Version: 20.10.18
 Storage Driver: overlay2
  Backing Filesystem: xfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Kernel Version: 5.15.0-1030-aws
 Operating System: Ubuntu 20.04.5 LTS
 OSType: linux
 Architecture: x86_64

Starting container cimg/ruby:2.7.5-browsers
Warning: No authentication provided, using CircleCI credentials for pulls from Docker Hub.
  image is cached as cimg/ruby:2.7.5-browsers, but refreshing...
2.7.5-browsers: Pulling from cimg/ruby
Digest: sha256:08391a7e5c03a4ac22234eee0487a312eb1991d1e70e228696a99a95dbd3cfda
Status: Image is up to date for cimg/ruby:2.7.5-browsers
cimg/ruby:2.7.5-browsers:
  using image cimg/ruby@sha256:08391a7e5c03a4ac22234eee0487a312eb1991d1e70e228696a99a95dbd3cfda
  pull stats: Image was already available so the image was not pulled
  time to create container: 616ms
Starting container cimg/postgres:13.5
Warning: No authentication provided, using CircleCI credentials for pulls from Docker Hub.
  image is cached as cimg/postgres:13.5, but refreshing...
13.5: Pulling from cimg/postgres
Digest: sha256:2de400e59bbba596f83b993f4c7922851bb06de22dae45c5184110d3fe65a70e
Status: Image is up to date for cimg/postgres:13.5
cimg/postgres:13.5:
  using image cimg/postgres@sha256:2de400e59bbba596f83b993f4c7922851bb06de22dae45c5184110d3fe65a70e
  pull stats: Image was already available so the image was not pulled
  time to create container: 60ms
Time to upload agent and config: 393.941236ms
Time to start containers: 516.542993ms

We are both running the same script (as you have posted above), but your build environment is Ubuntu 22.04.2, while mine is 20.04.5.

One key question - are you running a Local server install of CircleCI? As your config.yml does not have an executor defined I do not believe you are using a self-hosted runner.

5

Hey, thanks for your help! Appreciate it.

One key question - are you running a Local server install of CircleCI? As your config.yml does not have an executor defined I do not believe you are using a self-hosted runner.

Am not sure, I just installed the CircleCI cli tool and my understanding is that we are using docker as the executor?

From the documents on (docs/executor-intro/):

To access the Docker execution environment, use the docker executor and specify an image. For a full list of convenience images, which are built by CircleCI, see the CircleCI Developer Hub

jobs:
  build: # name of your job
    docker: # executor type
      - image: cimg/base:stable # primary container will run the latest, production-ready base image
        auth:
          username: mydockerhub-user
          password: $DOCKERHUB_PASSWORD  # context / project UI env-var reference

    steps:
        # Commands run in the primary container

Which is what we have on our config. Do I have to have something different if trying to re-create the docker environment on my local computer somehow?

Thanks again!

6

OK so you are running a job with a local command something like

circleci local execute JOB_NAME

And I guess your environment is a Ubuntu 22.04.2 installation.

I’ll be honest here, I’ve not worked with the local cli, all my builds are built around CircleCI’s AWS based instances and/or self-hosted instances that connect to CircleCI’s cloud-based service. So I may not be able to help much.

One thing that comes to mind is what version of docker do you have installed - Ubuntu distros have by default what is considered a ‘stable’ release of docker in their repos. The problem is that ‘stable’ can also mean dated.

7

Yup, exactly running through the command you just posted, but as per the documentation, since the 2.1 version onward you have to create a process.yml file like so:

circleci config process .circleci/config.yml > process.yml

And then run the actual job:

circleci local execute -c process.yml build

Interestingly, I downloaded the public CircleCI demo of a ruby on rails barebones app from githubDOTcom/CircleCI-Public/circleci-demo-ruby-rails and it was failing due to some versioning, I fixed the issues, changed the postgres image it was pointing to (using an older version from the cimage repo instead of the newer cimg) and I was able to create the aforementioned process.yml file and then run one of the two jobs on the docker file:

version: 2.1

orbs:
  ruby: circleci/ruby@1.1.0
  node: circleci/node@2

jobs:
  build:
    docker:
      - image: cimg/ruby:2.7-node
    steps:
      - checkout
      - ruby/install-deps
      # Store bundle cache
      - node/install-packages:
          pkg-manager: yarn
          cache-key: "yarn.lock"
  test:
    parallelism: 3
    docker:
      - image: cimg/ruby:2.7-node
      - image: cimg/postgres:13.5
        environment:
          POSTGRES_USER: circleci-demo-ruby
          POSTGRES_DB: rails_blog_test
          POSTGRES_PASSWORD: ""
    environment:
      BUNDLE_JOBS: "3"
      BUNDLE_RETRY: "3"
      PGHOST: 127.0.0.1
      PGUSER: circleci-demo-ruby
      PGPASSWORD: ""
      RAILS_ENV: test
    steps:
      - checkout
      - ruby/install-deps
      - node/install-packages:
          pkg-manager: yarn
          cache-key: "yarn.lock"
      - run:
          name: Wait for DB
          command: dockerize -wait tcp://localhost:5432 -timeout 1m
      - run:
          name: Database setup
          command: bundle exec rails db:schema:load --trace
      # Run rspec in parallel
      - ruby/rspec-test
      - ruby/rubocop-check

workflows:
  version: 2
  build_and_test:
    jobs:
      - build
      - test:
          requires:
            - build

Running the build job now executes properly with no errors, however, running the test job which requires the build one and has the postgres image dependency seems to fail so I’m thinking that creating containers in parallel is the problem here as running a job that only uses one image/container works fine as the build one here.

Still get the same error:

Status: Image is up to date for cimg/ruby:2.7-node
Creating Docker containers in parallel .Error:   error starting container cimg/postgres:13.5: Error response from daemon: invalid UTS mode: container91420d21a6aa20b7e81a5e782ba8b9428ad3e7364e73ef718aa19a270bf95c00

Error: 
Unexpected environment preparation error: Error response from daemon: invalid UTS mode: container91420d21a6aa20b7e81a5e782ba8b9428ad3e7364e73ef718aa19a270bf95c00

Step failed

Error: Unhandled prepare executor error: Error response from daemon: invalid UTS mode: container91420d21a6aa20b7e81a5e782ba8b9428ad3e7364e73ef718aa19a270bf95c00

Therefore I believe is something with multiple containers. I will try this modified file in Docker in my Windows box and see how it goes.

Just FYI, I do have the official Docker-Community repos installed on my Ubuntu 20.04 so they are indeed up to date:

Docker version 24.0.0, build 98fdcd7

Thanks a lot for your help!

8

Hello! I found this problem does not occurs on Docker Engine 20.10.
but following error was occured.

====>> Checkout code
Making checkout directory "/root/projects"
Copying files from "/tmp/_circleci_local_build_repo" to "/root/projects"
Error: fatal: detected dubious ownership in repository at '/tmp/_circleci_local_build_repo'
To add an exception for this directory, call:

        git config --global --add safe.directory /tmp/_circleci_local_build_repo

I think git is installed on circleci/picard image, so I have no way to fix it.

9

I’m experiencing the exact same problem on the pgcat repo (postgresml/pgcat). As you see on this repo, CircleCI works fine. But if I checkout this repo and try to run locally using these command:

circleci config process .circleci/config.yml > process.yml
circleci local execute -c process.yml build

It fails with this error output:

√ pgcat % circleci config process .circleci/config.yml > process.yml

√ pgcat % circleci local execute -c process.yml build               
Fetching latest build environment...
Docker image digest: sha256:008ba7f4223f1e26c11df9575283491b620074fa96da6961e0dcde47fb757014
====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 23.0.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Kernel Version: 5.15.49-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64

Starting container ghcr.io/postgresml/pgcat-ci:latest
  image is cached as ghcr.io/postgresml/pgcat-ci:latest, but refreshing...
ghcr.io/postgresml/pgcat-ci:latest:
  using image ghcr.io/postgresml/pgcat-ci@sha256:9097e06055cee3c25ebfcc7f71d3413e68125d1653ad1e31fcff0e7a49adcc67
  pull stats: Image was already available so the image was not pulled
  time to create container: 76ms
Starting container postgres:14
Starting container postgres:14
Starting container postgres:14
Starting container postgres:14
Starting container postgres:14
Error:   error starting container postgres:14: Error response from daemon: invalid UTS mode: containera4e4804657980605d031801469af669e540d8629ec7a0f4d108ab5434145ff58

Error:   error starting container postgres:14: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.41/containers/create": context canceled

Error:   error starting container postgres:14: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.41/containers/create": context canceled

Error:   error starting container postgres:14: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.41/containers/create": context canceled

Error:   error starting container postgres:14: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.41/containers/create": context canceled

latest: Pulling from postgresml/pgcat-ci
Digest: sha256:9097e06055cee3c25ebfcc7f71d3413e68125d1653ad1e31fcff0e7a49adcc67
Status: Image is up to date for ghcr.io/postgresml/pgcat-ci:latest
Error: 
Unexpected environment preparation error: Error response from daemon: invalid UTS mode: containera4e4804657980605d031801469af669e540d8629ec7a0f4d108ab5434145ff58

Step failed
Task failed
Error: Unhandled prepare executor error: Error response from daemon: invalid UTS mode: containera4e4804657980605d031801469af669e540d8629ec7a0f4d108ab5434145ff58
2 Likes
10

Hello,

I am also experiencing the same issue. I have tried various things that I found in random posts and CircleCI tickets. I have updated my PostgreSQL version, tried version 2.1 of the yml file, tried other Picard container versions as well. Always the same result.

We need to be able to run jobs locally in order to test them (as I am pretty sure everyone else does).

Environment:

Ubuntu 22.04
Docker Engine: 24.0.6
CircleCI CLI: 0.1.29314+148495a (latest at the moment)

System Information as printed when attempting to execute a job:

 Server Version: 24.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Kernel Version: 6.4.16-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64

My Docker container configuration is just this:

  docker:
    - image: circleci/python:3.8.10
    - image: cimg/postgres:16.1
11

Within your environment is the user account that executes the circlecil command in the defined Linux docker group?

The reason for asking is that “http://%2Fvar%2Frun%2Fdocker.sock/v1.41/containers/create” indicates communications with the docker socket and such access is controlled by the docker group.