"Too long with no output" while building Docker image


#1

We sporadically get the error Too long with no output (exceeded 10m0s) in the middle of building our image.

Setup:

- setup_remote_docker:
          reusable: true
- checkout
- run: docker-compose build --pull

#2

You can specify the no_output_timeout like this:

- run: 
    command: docker-compose build --pull
    no_output_timeout: 1200

#3

The problem is that our normal build image will finish in less than 5 minutes. With this sporadic failure, the build image process just seems to hang until it times out at 10 minutes. I don’t think increasing the timeout would fix anything.


#4

So I upped tried no_output_timeout with it set to 3600, and it still hangs and times out. This seems like something to do with the remote Docker host or network between the job runner and the remote host, but I’m not sure how to diagnose it.

As I mentioned before, a normal build with no cache should take less than 5 minutes. This hang only happens maybe ~5% of the time. However, since we use 4x parallel jobs, that translates to ~20% of our builds failing.

Any help you could provide, @rohara, would be appreciated.


#5

The only improvement I see you can make is to switch to the machine so at least connecting to Docker containers is consistent & reliable.


#6

The whole reason for us trying to use the Docker executor is the Docker image layer caching though.


#7

@sakai135 Can you PM me a build (or more than one) where you’re timing out with Docker? Our team wants to further investigate.


#8

I am seeing a similar problem on a Scala application. It fails hangs at the compile step of the application.

[success] Total time: 98 s, completed May 17, 2017 8:38:44 PM
> Too long with no output (exceeded 10m0s)

#9

It’s possible the thread is being killed. Are you restricting the memory usage of Scala?


#10

Switching back to sbt 0.13.11 from 0.3.14/15 fixed the hanging problem.


Run step ends with a bash shell causing step to timeout
#11

This isn’t really a solution. We need a later SBT to fix some assembly problems without having to muck around with merge strategies.


#12

@bhoggard Can you try this on 0.3.15?

cat /dev/null | sbt "test:test"

#13

This topic was automatically closed 41 days after the last reply. New replies are no longer allowed.