Local Checkout fails using CLI, tar terminates with signal 13

#1

Hello,

I recently updated my local build-agent to latest, and while trying to run my configuration locally I’m now getting an error on code checkout. Everything was working fine before updating the build-agent. Any ideas on how to solve this problem?

Configuration:

version: 2
jobs:
  build:
    working_directory: ~/overwatch.id/overwatchid-application/overwatchid-webapp/src/main/webapp
    docker:
      - image: circleci/node:10.15-browsers-legacy
    steps:
      - checkout:
          path: ~/overwatch.id

Output:

====>> Spin up Environment
Build-agent version 1.0.7998-a7024deb (2019-02-22T13:37:52+0000)
Starting container circleci/node:10.15-browsers-legacy
  using image circleci/node@sha256:b6d94197ff595bc1b544220815245e51e8f110116c21c67e64ba68f34198e675

Using build environment variables:
  BASH_ENV=/tmp/.bash_env-localbuild-1551203450
  CI=true
  CIRCLECI=true
  CIRCLE_BRANCH=service-providers-upsert
  CIRCLE_BUILD_NUM=
  CIRCLE_JOB=build
  CIRCLE_NODE_INDEX=0
  CIRCLE_NODE_TOTAL=1
  CIRCLE_REPOSITORY_URL=https://wescopeland_@bitbucket.org/overwatchid/overwatch.id.git
  CIRCLE_SHA1=203c018f09d101fa6150934cd3330fa764cb09a1
  CIRCLE_SHELL_ENV=/tmp/.bash_env-localbuild-1551203450
  CIRCLE_WORKING_DIRECTORY=~/overwatch.id/overwatchid-application/overwatchid-webapp/src/main/webapp

====>> Checkout code
  #!/bin/bash -eo pipefail
mkdir -p /home/circleci/overwatch.id && cd /tmp/_circleci_local_build_repo && git ls-files -z | xargs -0 tar -c | tar -x -C /home/circleci/overwatch.id
xargs: tar: terminated by signal 13
Error: Exited with code 125
Step failed
1 Like
#2

I’m not sure if this is relevant, but:

Does this path exist? I don’t usually specify the path at all and just use whatever the default ends up being, I think its a much simpler approach rather than having to keep track of the current working directory.

Could you try to remove working_directory and the path argument? What happens then?

#3

Hi @wlcopeland, currently I’m facing the same issue running circleci local. Have you found any solution?

Best,

#4

I’m facing the same problem now. @ls_1212 @wlcopeland did you find any solution?

#5

Hi folks. @ls_1212 , @saveman71 couple questions, please. :slight_smile:

  • What OS are you running?
  • What does tar --version say?
  • Are you on the latest version of the CLI?

Also, to answer Lev’s question, do you get this error without a path: to Checkout?

#6

Hi @saveman71, I haven’t found the solution yet… I’ll post if I found it :+1:

#7

Hi @drazisil, thanks for your contact…

  • I’m using macOS Mojave v10.14.4
  • In the terminal, tar --version: bsdtar 2.8.3 - libarchive 2.8.3
  • Yes, I’m using the latest
#8

I’m running an updated arch linux install.

❯ tar --version
tar (GNU tar) 1.32
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by John Gilmore and Jay Fenlason.

My cli should be the latest version (circleci update) just exits with “Done”.

And my checkout step doesn’t have a path:, just checkout.

Relevant logs:

====>> Checkout code
  #!/bin/bash -eo pipefail
mkdir -p /home/circleci/project_name && cd /tmp/_circleci_local_build_repo && git ls-files -z | xargs -0 tar -c | tar -x -C /home/circleci/project_name && cp -a /tmp/_circleci_local_build_repo/.git /home/circleci/project_name                                                
xargs: tar: terminated by signal 13
Error: Exited with code 125
Step failed
#9

We can see the command being run here. Does this work if you try it manually? Two different OS versions, two different tar versions. If this were a issue with withe the CLI in general we would be getting a lot more reports, but it seems there is some form of edge case you both are hitting.

The more things we try, the more we can isolate the bug report and figure out a fix.

#10

I can indeed (for now) locate the issue to the following command:

git ls-files -z | xargs -0 tar -c | tar -t -C ~/tmp-circleci

Signal 13 means the second tar is still reading and the first would be finished writing. (cf https://stackoverflow.com/a/27903078/2367848, which I guess you read as well)

Edit: this seems to work with a test repository with only 1 file (committed). The copy happens w/o error. Maybe a problematic file?

1 Like
#11

Or maybe a really large one? Will have to narrow it down by removing files, I’m afraid. Might be a good use case for https://git-scm.com/docs/git-bisect to aid in troubleshooting sanity

#12

Unfortunately I’m not willing to bisect a 25k commit codebase :smiley:

I think I have a lead, the xargs -0 doesn’t seem to be working. If I had -t to xargs to print the commands, I’m only shown two big tar calls, the first with the first 2994 files (counted the number of spaces + 1), and the second one with 732 files.

git ls-files|wc -l yields 3728 (so i’m missing 2 files, but maybe I’m not counting right)

1 Like
#13

:mal_speechless: I…don’t blame you!

Hope you can get it figured out and would love if you can share when you do, so we can make a note for others.

#14

I’ve isolated the problem even more:

It’s actually the number of files in the repo that triggers the error. Apparently 3700 files is too much :sweat_smile:

git ls-files -z | xargs -L 1 -t -0 tar -c | tar -x -C ~/tmp-circleci

will fail reliably for every repo, by limiting the number of files per xargs call to 1, thus making more than 1 call to tar, which is apparently not supported when piping.

1 Like
#15

Removing xargs from the chain and using tar’s built-in -T option works!

git ls-files -z | tar --null -T - -c | tar -x -C ~/tmp-circleci

I don’t know how portable that is. (e.g. on @ls_1212’s mac?)

git ls-files | tar -T - -c | tar -x -C ~/tmp-circleci

Should work as well and is maybe more portable across tar versions

3 Likes
#16

Thank you so much for that great research and writeup. I’ll get a bug filed. (wonders what % of our customers have that many files AND do local checkouts) :thinking:

1 Like
#17

No problem! :wink:

Could you bring light to how it used to work before? Did @wlcopeland just happen to reach that limit of files and thought it was an update to the CLI that triggered the error, or was the way to transfer files really changed and used to work with a large number of files?

Side note, the xargs limit is system dependent, so maybe the other users ran into different limits, hard to know.

#18

It appears to have been added 7 months ago, to fix an issue where the local CLI was copying everything, and not just files that a checkout would.

So it looks due to @wlcopeland updating that triggered it for them.

#20

Alright seems fair! Weird indeed that this few users reported it. Went ahead and created a recap here: https://stackoverflow.com/a/55906637/2367848

2 Likes
#21

We’re running into this as well…