Cannot install PHP 7.1 on `machine` because of locked /var/lib/dpkg/lock file

I am trying to use machine and install PHP 7.1 on it but I am running into problems with files being locked after running sudo apt-get update. These are the commands I am using:

sudo apt-add-repository ppa:ondrej/php
sudo apt-get update
sudo apt-get install php7.1

I know this may be considered a Linux issue, but this is such a basic use-case that I would hope CircleCI would want it to be possible without having to do resort to complex Bash scripting to get around this problem.

BTW, I am using machine because I later want to my own set of Docker containers that require a sandboxed VM to operate correctly.

Would you add here the full logs from these apt commands?

#!/bin/bash -eo pipefail
cat /tmp/deploy.log
===[ Adding Apt-Get repository for PHP ]==============================
gpg: keyring `/tmp/tmp1xy_7j_t/secring.gpg' created
gpg: keyring `/tmp/tmp1xy_7j_t/pubring.gpg' created
gpg: requesting key E5267A6C from hkp server keyserver.ubuntu.com
gpg: /tmp/tmp1xy_7j_t/trustdb.gpg: trustdb created
gpg: key E5267A6C: public key "Launchpad PPA for Ondřej Surý" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
OK
===[ Updating Apt-Get ]==============================
Ign:1 http://dl.google.com/linux/chrome/deb stable InRelease
Hit:2 http://dl.google.com/linux/chrome/deb stable Release
Hit:3 http://security.ubuntu.com/ubuntu xenial-security InRelease
Hit:4 http://ppa.launchpad.net/git-core/ppa/ubuntu xenial InRelease
Hit:5 http://archive.canonical.com/ubuntu xenial InRelease
Hit:6 http://archive.ubuntu.com/ubuntu xenial InRelease
Hit:7 http://archive.ubuntu.com/ubuntu xenial-updates InRelease
Get:9 http://ppa.launchpad.net/ondrej/php/ubuntu xenial InRelease [23.9 kB]
Hit:10 http://archive.ubuntu.com/ubuntu xenial-backports InRelease
Hit:11 http://ppa.launchpad.net/openjdk-r/ppa/ubuntu xenial InRelease
Hit:12 https://download.docker.com/linux/ubuntu xenial InRelease
Hit:13 https://cli-assets.heroku.com/apt ./ InRelease
Get:14 http://ppa.launchpad.net/ondrej/php/ubuntu xenial/main amd64 Packages [51.1 kB]
Hit:15 https://packagecloud.io/circleci/trusty/ubuntu xenial InRelease
Get:16 http://ppa.launchpad.net/ondrej/php/ubuntu xenial/main Translation-en [28.1 kB]
Fetched 103 kB in 1s (58.6 kB/s)
Reading package lists...
===[ Checking for processes locking ]==============================
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
dpkg    6183 root    3uW  REG    8,1        0 57773 /var/lib/dpkg/lock
root      2759  0.0  0.0   4504   756 ?        Ss   20:07   0:00 /bin/sh /usr/lib/apt/apt.systemd.daily install
root      2773  0.0  0.0   4504  1652 ?        S    20:07   0:00 /bin/sh /usr/lib/apt/apt.systemd.daily lock_is_held install
root      6183 34.7  0.4  58164 37428 pts/2    Rs+  20:08   0:03 /usr/bin/dpkg --force-confnew --status-fd 10 --unpack --auto-deconfigure /var/cache/apt/archives/libxml2-dev_2.9.3+dfsg1-1ubuntu0.6_amd64.deb /var/cache/apt/archives/libxml2_2.9.3+dfsg1-1ubuntu0.6_amd64.deb /var/cache/apt/archives/bsdtar_3.1.2-11ubuntu0.16.04.4_amd64.deb /var/cache/apt/archives/libarchive13_3.1.2-11ubuntu0.16.04.4_amd64.deb /var/cache/apt/archives/libavresample-ffmpeg2_7%3a2.8.15-0ubuntu0.16.04.1_amd64.deb /var/cache/apt/archives/libswresample-ffmpeg1_7%3a2.8.15-0ubuntu0.16.04.1_amd64.deb /var/cache/apt/archives/libpostproc-ffmpeg53_7%3a2.8.15-0ubuntu0.16.04.1_amd64.deb /var/cache/apt/archives/libswscale-ffmpeg3_7%3a2.8.15-0ubuntu0.16.04.1_amd64.deb /var/cache/apt/archives/libavformat-ffmpeg56_7%3a2.8.15-0ubuntu0.16.04.1_amd64.deb /var/cache/apt/archives/libavfilter-ffmpeg5_7%3a2.8.15-0ubuntu0.16.04.1_amd64.deb /var/cache/apt/archives/libavdevice-ffmpeg56_7%3a2.8.15-0ubuntu0.16.04.1_amd64.deb /var/cache/apt/archives/libavcodec-ffmpeg56_7%3a2.8.15-0ubuntu0.16.04.1_amd64.deb /var/cache/apt/archives/libavutil-ffmpeg54_7%3a2.8.15-0ubuntu0.16.04.1_amd64.deb /var/cache/apt/archives/ffmpeg_7%3a2.8.15-0ubuntu0.16.04.1_amd64.deb /var/cache/apt/archives/libxcursor1_1%3a1.1.14-1ubuntu0.16.04.2_amd64.deb /var/cache/apt/archives/linux-modules-4.15.0-1017-gcp_4.15.0-1017.18~16.04.1_amd64.deb /var/cache/apt/archives/linux-image-4.15.0-1017-gcp_4.15.0-1017.18~16.04.1_amd64.deb /var/cache/apt/archives/linux-gcp_4.15.0.1017.29_amd64.deb /var/cache/apt/archives/linux-image-gcp_4.15.0.1017.29_amd64.deb /var/cache/apt/archives/linux-gcp-headers-4.15.0-1017_4.15.0-1017.18~16.04.1_amd64.deb /var/cache/apt/archives/linux-headers-4.15.0-1017-gcp_4.15.0-1017.18~16.04.1_amd64.deb /var/cache/apt/archives/linux-headers-gcp_4.15.0.1017.29_amd64.deb /var/cache/apt/archives/linux-libc-dev_4.4.0-134.160_amd64.deb /var/cache/apt/archives/openjdk-8-jdk_8u181-b13-0ubuntu0.16.04.1_amd64.deb /var/cache/apt/archives/openjdk-8-jdk-headless_8u181-b13-0ubuntu0.16.04.1_amd64.deb /var/cache/apt/archives/openjdk-8-jre_8u181-b13-0ubuntu0.16.04.1_amd64.deb /var/cache/apt/archives/openjdk-8-jre-headless_8u181-b13-0ubuntu0.16.04.1_amd64.deb /var/cache/apt/archives/libav-tools_7%3a2.8.15-0ubuntu0.16.04.1_all.deb
===[ Installing PHP 7.1 ]==============================
E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)
E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
flock: getting lock took 0.000006 seconds
flock: executing /bin/bash
===[ Display PHP version info ]==============================
/home/circleci/repo/.circleci/install-php.sh: line 50: php: command not found

The section entitled “Checking for processes locking” used these commands:

sudo lsof /var/lib/dpkg/lock
sudo ps aux | grep -e apt -e adept -e dpkg | grep -v grep

I can’t replicate that, it works for me:

Final output:

#!/bin/bash -eo pipefail
php -v

PHP 7.1.20-1+ubuntu14.04.1+deb.sury.org+1 (cli) (built: Jul 25 2018 10:32:41) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.1.0, Copyright (c) 1998-2018 Zend Technologies
    with Zend OPcache v7.1.20-1+ubuntu14.04.1+deb.sury.org+1, Copyright (c) 1999-2018, by Zend Technologies

Would you put your config here?

Hmm, it looks like you have other processes doing system updates at the same time.

Here is config.yml:

version: 2
jobs:
    build:
        machine:
            image: circleci/classic:201808-01
        working_directory: ~/repo
        environment:
        steps:
            - checkout
            - run: bash .circleci/install-php.sh

And here is install-php.sh:

#!/usr/bin/env bash
sudo apt-add-repository -y ppa:ondrej/php
sudo apt-get update
sudo apt-get install php7.1
php -v

SOMETIMES it works. Sometimes it fails because of lock.

Aha! A quick bit of duck-duck-going lead to this solution for someone having the same problem. Looks like it is a problem with Systemd generally.

I don’t think I have seen this reported here before. I wonder whether people would normally install things like PHP first, and then do their checkout. You have them reversed, which perhaps gave the machine more time to throw a spanner in the works. :smirk_cat:

So here is my results from queuing up a bunch. One success and 9 failures.

I will try with checkout after and let you know (after I get back from some dinner. :-))

Is that even with the stop, kill and wait? I can’t imagine why that would fail. (Of course it depends on what failed - were these failures still lock issues?)

How long does your checkout take? I just added a 60 second sleep, and it still works. Bah!

@halfer

I have decided to give up on this and take the path of least resistance for now and just use a Docker image. My thoughts are now that CircleCI are putting no efforts into machine so choosing it will be folly. I guess I will have to figure out Docker-in-Docker later.

Anything, big thanks for all your efforts.

I suspect that DinD is more work, to be honest. My view is that if you have a PHP build to run, this is your best bet to meet the 2.0 deadline, especially if you are not familiar with Docker. I’d recommend:

  • Posting the result of swapping the install and checkout
  • Posting the build logs after the kill/wait script is installed

I am entirely confident that this can be made to work with minimal extra effort.

Hi @halfer - Thanks for the follow up.

Sorry, I was probably not clear on my short term requirements. I don’t need to run other Docker containers inside a VM before August 31st, that is longer term. I was just trying to kill two birds with one stone and get experience with machine. I can cross the D-in-D bridge or machine bridge later when we actually get to that point.

As it is, I have already moved on and have been working on the scripts I need to do the deploys that we are using CircleCI to make happen. I’ve still got a lot of work to do there so I don’t think it would be wise at this time to try to resolve the machine + PHP install problems. But I really thank you for your interest in helping.

Maybe when I revisit this (or someone else will have solved the problem by then?)

running into a similar issue when installing awscli on machine classic:201808-01. but only some of the branches have this error. unfortunately those branches are master and release. all of the other development branches skip right through without any problem. i’ve tried throwing a sudo apt-get install -f to force dpkg to resolve its issues and unlock, but that only seems to work on the development branches and not master.

this issue only started to show up when we moved from machine classic:201710-01 to the more recent image. i’m thinking now there’s something wrong with that machine.

@worc: would you add a public and reproducible example on GitHub?

i can’t just put internal code up on a public repo, but i can at least post the makefile command that’s falling apart. maybe there’s a simple bug i just can’t see?

i’m already seeing one issue with the hack/workaround missing the double-ampersand to block while it executes.

.PHONY: ci_setup
ci_setup: ## Setup the travis environmnet
	@if [[ -n "$$BUILD_ENV" ]] && [[ "$$BUILD_ENV" == "test" ]]; then echo -e "$(INFO_COLOR)THIS IS EXECUTING AGAINST THE TESTING ENVIRONMEMNT$(NO_COLOR)"; fi
	@echo "Installing AWS cli"
	@pip install --user awscli && \
		export PATH=$PATH:$HOME/.local/bin
	@echo "Installing Chrome"
	@curl -L -o google-chrome.deb https://s3.amazonaws.com/<>/files/chrome/chrome64_65.0.3325.181.deb && \
		sudo apt-get install -f
		sudo dpkg -i google-chrome.deb && \
		sudo sed -i 's|HERE/chrome\"|HERE/chrome\" --disable-setuid-sandbox|g' /opt/google/chrome/google-chrome && \
		rm google-chrome.deb
	@echo "Configuring Node & Yarn"
	@sudo rm -rf /opt/circleci/.nvm && \
		curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add - && \
		echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list && \
		curl -sS https://deb.nodesource.com/gpgkey/nodesource.gpg.key | sudo apt-key add - && \
		echo "deb https://deb.nodesource.com/node_10.x xenial main" | sudo tee /etc/apt/sources.list.d/nodesource.list && \
		sudo apt-get update -y && \
		sudo apt-get install -y nodejs yarn

My last comment on this thread.

No, but you can clone the repo and then cut it down to such a bare-bones example that there are no intellectual property issues. Help people help you. :slightly_smiling_face:

Also, see the solution I linked to in August up-thread.

the barebones is already there in the makefile…

This seems still to be an issue and happens with image: circleci/classic:201808-01

It does not happen with image: circleci/classic:201710-01

version: 2
jobs:
  build:
    working_directory: ~/code
    machine:
      enabled: true
      image: circleci/classic:201808-01
      #image: circleci/classic:201710-01
  
    steps: 
      - run:
          name: Install Expect
          command: |
            sudo apt-get update
            sudo apt-get install -f expect
            expect -v

Error:

E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)
E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
Exited with code 100

the issue was bad enough and intractable enough that we’ve moved off circleci and onto travis ci where machine setups are supported.

the solutions here go down a path of fixing the VM with special commands and scripts and we really don’t want to be in the businesses of managing our managed service.

This works fine for me. I’ve added a sleep in, I think that might have been one of the fixes linked to earlier. Has your experience been intermittent, or has it failed reliably?

That’s fine - I don’t work for CircleCI, so I have no axe to grind regarding whose service you use. I am also a big believer in letting teams choose their own tools. Nevertheless, you are likely to hit a roadbump with Travis, at which point you’ll take out your frustration on Travis forum volunteers and decide that only GitLab cares about support. Eventually you’ll run out of (good) CI providers to try.

For your own benefit, I would urge you to change your approach.