Git LFS broken in recent Ubuntu 20.04 machine images (arm64)

dennisameling · January 4, 2023, 12:32pm

I’m running into a strange issue when using newer ubuntu-2004 images. Git LFS stops working. Here’s a failing CI run caused by Git LFS not being triggered correctly: https://app.circleci.com/pipelines/github/dennisameling/signal-desktop-automation/56/workflows/be3fde3e-0507-4746-89ac-1c511294c149/jobs/32

The log below is from the older ubuntu-2004:202101-01 image which is working correctly (note that deps/sqlcipher.tar.gz is being downloaded correctly).

Successful job URL: https://app.circleci.com/pipelines/github/dennisameling/signal-desktop-automation/69/workflows/d7299acc-527d-46c9-a2ee-c7dc4587bbc1/jobs/43

$ sudo apt-get install git-lfs
$ git lfs install
$ which git-lfs
/usr/bin/git-lfs
$ git-lfs --version
git-lfs/2.9.2 (GitHub; linux arm64; go 1.13.5)
$ git clone https://github.com/signalapp/better-sqlite3
$ cd better-sqlite3
$ GIT_TRACE=1 git checkout afdbd49cf18d0046d7385ca4d23da20c66f00d50
11:48:36.080725 git.c:444               trace: built-in: git checkout afdbd49cf18d0046d7385ca4d23da20c66f00d50
11:48:36.094073 run-command.c:664       trace: run_command: 'git-lfs filter-process'
11:48:36.116993 trace git-lfs: exec: git 'version'
11:48:36.134460 trace git-lfs: exec: git 'config' '-l'
11:48:36.142263 trace git-lfs: Install hook: pre-push, force=false, path=/home/circleci/better-sqlite3/.git/hooks/pre-push
11:48:36.142375 trace git-lfs: Install hook: post-checkout, force=false, path=/home/circleci/better-sqlite3/.git/hooks/post-checkout
11:48:36.142442 trace git-lfs: Install hook: post-commit, force=false, path=/home/circleci/better-sqlite3/.git/hooks/post-commit
11:48:36.142505 trace git-lfs: Install hook: post-merge, force=false, path=/home/circleci/better-sqlite3/.git/hooks/post-merge
11:48:36.142551 trace git-lfs: Initialize filter-process
11:48:36.142706 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
11:48:36.159520 trace git-lfs: tq: running as batched queue, batch size of 100
11:48:36.160255 trace git-lfs: filepathfilter: accepting "deps/sqlcipher.tar.gz"
11:48:36.246067 trace git-lfs: tq: sending batch of size 1
11:48:36.246847 trace git-lfs: api: batch 1 files
11:48:36.247102 trace git-lfs: HTTP: POST https://github.com/signalapp/better-sqlite3.git/info/lfs/objects/batch
11:48:36.488328 trace git-lfs: HTTP: 200
11:48:36.490373 trace git-lfs: HTTP: {"objects":[{"oid":"de494ea3699cb0ac424b11870c7b2a780249f8755215c22ec7616d7dc12b4b3f","size":41118646,"actions":{"download":{"href":"https://github-cloud.githubusercontent.com/alambic/media/345736527/de/49/de494ea3699cb0ac424b11870c7b2a780249f8755215c22ec7616d7dc12b4b3f?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMWPLRQEC4XCWWPA%2F20230104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230104T114836Z&X-Amz-Expires=3600&X-Amz-Signature=2fe370022d99524e16e8f0febecb20edf3b4e143d35175ccedc1906efddd3d2b
11:48:36.492911 trace git-lfs: HTTP: &X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=355014760&token=1","expires_at":"2023-01-04T12:48:36Z","expires_in":3600}}}]}
11:48:36.493018 trace git-lfs: tq: starting transfer adapter "basic"
11:48:36.495824 trace git-lfs: HTTP: GET https://github-cloud.githubusercontent.com/alambic/media/345736527/de/49/de494ea3699cb0ac424b11870c7b2a780249f8755215c22ec7616d7dc12b4b3f
11:48:36.565256 trace git-lfs: HTTP: 200
11:48:37.018428 trace git-lfs: filepathfilter: accepting "deps/sqlcipher.tar.gz"
Note: switching to 'afdbd49cf18d0046d7385ca4d23da20c66f00d50'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at afdbd49 Fix build with Electron 21-x-y
11:48:37.215196 run-command.c:664       trace: run_command: .git/hooks/post-checkout 835217678aaaefaa7992f5d3711eb2a19ec1232f afdbd49cf18d0046d7385ca4d23da20c66f00d50 1
11:48:37.234732 git.c:730               trace: exec: git-lfs post-checkout 835217678aaaefaa7992f5d3711eb2a19ec1232f afdbd49cf18d0046d7385ca4d23da20c66f00d50 1
11:48:37.235593 run-command.c:664       trace: run_command: git-lfs post-checkout 835217678aaaefaa7992f5d3711eb2a19ec1232f afdbd49cf18d0046d7385ca4d23da20c66f00d50 1
11:48:37.247741 trace git-lfs: exec: git 'version'
11:48:37.274973 trace git-lfs: exec: git 'config' '-l'
11:48:37.282731 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
11:48:37.302073 trace git-lfs: NewLsFiles: running in /home/circleci/better-sqlite3 git ls-files -z --others --cached --exclude-standard
11:48:37.311315 trace git-lfs: findAttributeFiles: located .gitattributes
11:48:37.312355 run-command.c:44        trace: run_command: running exit handler for pid 9932
11:48:37.312541 trace git-lfs: filepathfilter: rewrite ".git" as "**/.git/**"
11:48:37.315083 trace git-lfs: filepathfilter: rewrite "**/.git" as "**/.git"
11:48:37.315230 trace git-lfs: filepathfilter: accepting "tmp"

Newer image versions like ubuntu-2004:202201-02 seem to come with git-lfs preinstalled in /usr/local/bin/git-lfs, so I didn’t do sudo apt-get install git-lfs, but the preinstalled version isn’t downloading deps/sqlcipher.tar.gz:

$ git lfs install
$ which git-lfs
/usr/local/bin/git-lfs
$ git-lfs --version
git-lfs/3.1.1 (GitHub; linux arm64; go 1.17.6)
$ git clone https://github.com/signalapp/better-sqlite3
$ cd better-sqlite3
$ GIT_TRACE=1 git checkout afdbd49cf18d0046d7385ca4d23da20c66f00d50
11:50:41.255873 git.c:458               trace: built-in: git checkout afdbd49cf18d0046d7385ca4d23da20c66f00d50
11:50:41.272002 run-command.c:654       trace: run_command: 'git-lfs filter-process'
11:50:41.284366 trace git-lfs: exec: git 'version'
11:50:41.291889 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir' '--show-toplevel'
11:50:41.295262 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
11:50:41.298008 trace git-lfs: exec: git 'config' '--includes' '--local' 'lfs.repositoryformatversion'
11:50:41.300822 trace git-lfs: exec: git 'config' '--includes' '--replace-all' 'lfs.repositoryformatversion' '0'
11:50:41.305018 trace git-lfs: exec: git 'config' '--includes' '-l'
11:50:41.312381 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
11:50:41.316184 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' ':.lfsconfig'
11:50:41.322129 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' 'HEAD:.lfsconfig'
11:50:41.334896 trace git-lfs: Install hook: pre-push, force=false, path=/home/circleci/better-sqlite3/.git/hooks/pre-push
11:50:41.334984 trace git-lfs: Install hook: post-checkout, force=false, path=/home/circleci/better-sqlite3/.git/hooks/post-checkout
11:50:41.335019 trace git-lfs: Install hook: post-commit, force=false, path=/home/circleci/better-sqlite3/.git/hooks/post-commit
11:50:41.335051 trace git-lfs: Install hook: post-merge, force=false, path=/home/circleci/better-sqlite3/.git/hooks/post-merge
11:50:41.335086 trace git-lfs: Initialize filter-process
11:50:41.338669 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
11:50:41.362369 trace git-lfs: tq: running as batched queue, batch size of 100
Note: switching to 'afdbd49cf18d0046d7385ca4d23da20c66f00d50'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at afdbd49 Fix build with Electron 21-x-y
11:50:41.449684 run-command.c:654       trace: run_command: .git/hooks/post-checkout 835217678aaaefaa7992f5d3711eb2a19ec1232f afdbd49cf18d0046d7385ca4d23da20c66f00d50 1
11:50:41.455892 git.c:745               trace: exec: git-lfs post-checkout 835217678aaaefaa7992f5d3711eb2a19ec1232f afdbd49cf18d0046d7385ca4d23da20c66f00d50 1
11:50:41.455929 run-command.c:654       trace: run_command: git-lfs post-checkout 835217678aaaefaa7992f5d3711eb2a19ec1232f afdbd49cf18d0046d7385ca4d23da20c66f00d50 1
11:50:41.460157 trace git-lfs: exec: git 'version'
11:50:41.463951 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir' '--show-toplevel'
11:50:41.469151 trace git-lfs: exec: git 'config' '--includes' '-l'
11:50:41.472858 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
11:50:41.476167 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' ':.lfsconfig'
11:50:41.479456 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' 'HEAD:.lfsconfig'
11:50:41.482434 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
11:50:41.485390 trace git-lfs: NewLsFiles: running in /home/circleci/better-sqlite3 git ls-files -z --cached --exclude-standard --others
11:50:41.486333 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'ls-files' '-z' '--cached' '--exclude-standard' '--others'
11:50:41.492237 trace git-lfs: findAttributeFiles: located .gitattributes
11:50:41.493546 run-command.c:45        trace: run_command: running exit handler for pid 9529
11:50:41.493723 trace git-lfs: filepathfilter: creating pattern ".git" of type gitignore
11:50:41.493765 trace git-lfs: filepathfilter: creating pattern "**/.git" of type gitignore
11:50:41.493813 trace git-lfs: filepathfilter: accepting "tmp"

I also tried removing the preinstalled /usr/local/bin/git-lfs and installing through sudo apt-get install git-lfs, to no avail. Does CircleCI add some specific Git LFS config that would explain why it’s not working on newer Ubuntu images?

owenjoliver · February 1, 2023, 5:41pm

Hello

Thank you for providing that build link I can see you were able to resolve this yourself would it be possible to tell us how you were able to do this incase other people encounter the same issue?

Kind Regards
Owen Oliver

dennisameling · February 10, 2023, 8:51am

Thanks for your reply. I wasn’t able to resolve it and as mentioned in the topic description, I downgraded to ubuntu-2004:202101-01 which works just fine. Here’s the pipeline config:

github.com

dennisameling/signal-desktop-automation/blob/main/.circleci/config.yml

version: 2.1

parameters:
  ref_to_build:
    type: string
    default: ""
  github_actions_release_upload_url:
    type: string
    default: ""

jobs:
  build:
    machine:
      # Fixed to this version due to issues with Git LFS
      # https://discuss.circleci.com/t/git-lfs-broken-in-recent-ubuntu-20-04-machine-images-arm64/46473/1
      image: ubuntu-2004:202101-01
    resource_class: arm.medium
    steps:
      - run:
          name: "Setup environment variables"

This file has been truncated. show original

dennisameling · February 24, 2024, 2:45pm

Just updated the image from ubuntu-2004:202101-01 to ubuntu-2004:2023.07.1 and didn’t run into the git-lfs issue this time great stuff!

system · March 2, 2024, 2:45pm

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Is there any way to disable git-lfs in Ubuntu 14.04 (Trusty) image? Build Environment git-lfs	15	7981	June 18, 2018
Xcode 11.2 macOS Catalina missing git lfs Build Environment git-lfs , xcode	4	3006	December 6, 2019
Error when running tests with Git LFS Build Environment	0	843	January 5, 2022
Cannot pull with git-lfs Feedback & Bug Reports git-lfs , git	2	2311	June 18, 2018
Ubuntu 14.04 Build Image Update 201704-01 Announcements build-image	2	3786	June 18, 2018

Git LFS broken in recent Ubuntu 20.04 machine images (arm64)

Related topics