Circle CI v2 and Android - Memory issues?

android
docker
gradle
2.0

#1

I’m trying to convert my Android project from Circle CI v1 to v2.
My v1 configuration can be found here:

My v1 configuration is here: https://github.com/AnySoftKeyboard/AnySoftKeyboard/blob/1.8-r9/circle.yml
I don’t have any major issues with it - it works well. But slow.

My v2 migration PR is here: https://github.com/AnySoftKeyboard/AnySoftKeyboard/pull/936

I tried to make it as close as possible to the previous configuration, and then improve on it with V2 stuff.

But, to my problem, right now the Gradle executor is dying during the test phase with error org.gradle.process.internal.ExecException: Process 'Gradle Test Executor 1' finished with non-zero exit value 137 (for example, this run https://circleci.com/gh/AnySoftKeyboard/AnySoftKeyboard/313).

From what I understand, this is a out-of-memory problem. I know that Docker will kill processes that exceed the memory quota. So, I tried to limit Gradle to use much less memory (512MB) than I, successfully, used in my v1 configuration (1600MB). That didn’t work either. so, I might be wrong about my out-of-memory guess.

Any leads here?


Android unit test fails circleci 2.0
#2

Might be related: http://matthewkwilliams.com/index.php/2016/03/17/docker-cgroups-memory-constraints-and-java-cautionary-tale/


#3

Maybe this? https://developers.redhat.com/blog/2017/03/14/java-inside-docker/


#4

It’s possibly a few things but we haven’t gotten Android builds running in docker yet. We’ve been focused on getting it running in the machine. And with that said, I want to make it clear we know that’s far less than ideal and we’re still working on a solution vs a workaround.


#5

then what are you suggesting? That’s I’ll switch to machine executor? Can you provide some example for that executor?


#6

I recommend holding off on Android until we publish official docs. It’s really not a fun process and we still don’t have a green build that we can show off. We have a lot of hours into it and I can’t in good conscience advise you to squander that time.


#7

@rohara - Following our conversations and workarounds of last week I’ve just managed to get my first green build of my Android app on CircleCI 2.0!

Want to show that one off?

:blush:

Here’s my circle.yml

version: 2
jobs:
  build:
    docker:
      - image: bitriseio/docker-android
    working_directory: ~/OceanLife
    steps:
      - checkout
      - run:
          name: Assemble & Test (Amazon Flavour)
          command: ./gradlew assembleWithAmazon testWithAmazonDebugUnitTest
          environment:
            TERM: dumb
      - run:
          name: Assemble & Test (Google Flavour)
          command: ./gradlew assembleWithGoogle testWithGoogleDebugUnitTest
          environment:
            TERM: dumb

Proof!


#8

@BrantApps I am DEFINITELY showing this off internally.


#9

I got a green build with Docker as well:
https://circleci.com/gh/AnySoftKeyboard/AnySoftKeyboard/330

The issue was Docker terminating Gradle executors because they consume too much memory.
This is because Java runtime is unaware of cgroups limitations, so I had to be explicit about the memory limitations by adding the environment variable _JAVA_OPTIONS: "-Xmx1024m"

@rohara, what is the Docker memory quota? And can we have that as an environment variable (as well as the CPU count).


#10

I will open a feature request. Two cores, four gigabytes of RAM.


#11

The most I could get was: _JAVA_OPTIONS: "-Xmx1500m -XX:ParallelGCThreads=2 -XX:ConcGCThreads=2 -XX:ParallelGCThreads=2 -Djava.util.concurrent.ForkJoinPool.common.parallelism=2"
But that might be because I allow two instances of the JVM.

Anyhow, thanks a lot for the help. I’m happy with the speed boost.

I’m going to add deploy next.


#12

Any updated on this topic?
We have run into the same issue. Unless we set the heap to something lower than 1/4 of the container memory (1024m in this case), our java processes are killed with a 137 exit code. So at the moment we are using _JAVA_OPTIONS: "-Xmx1024m" but this is really not enough for Gradle when building a large Android project. Given that the Circle 2 containers are 4Gb, we should be able to use a heap much larger than 1Gb.


#13

I’m not sure if this is relevant to this specific case, but I was having a similar issue (my build is not Android, but is running Gradle in CircleCI v2 Docker container; the process is being killed with exit code 137). Limiting memory did not solve the problem in my case.

I discovered that Gradle was running with 32 workers, which by far exceeds the 2 CPUs assigned to the container. Adding the --info flag to the end of the Gradle command confirms this. In my case, I changed the following command:
./gradlew test --no-daemon
to
./gradlew test --no-daemon --max-workers 2

Hopefully this information helps someone if they are encountering similar issues.


#14

Yes, that was also required for me. I passed the Max threads count as a JVM environment variable and also set it in the gradle.properties file


#15

I’ve tried all the suggestions from this thread but I still see the occasional Process ‘Gradle Test Executor 1’ finished with non-zero exit value 137 error, and I see more frequent Too long with no output (exceeded 10m0s) errors that I can’t figure out, but I assume something is hanging/deadlocking because of the memory restrictions. The only suggestion from this thread I haven’t tried is setting the max threads in gradle,properties, bc I couldn’t figure out what key to use. Menny, can you post that?

FWIW, here’s my circle config snippet for my app (i’m in a mono-repo, hence the pwd noise):

android-consumer:
  working_directory: ~/recharge
  docker:
    - image: circleci/android:api-26-alpha
  environment:
    # from https://discuss.circleci.com/t/circle-ci-v2-and-android-memory-issues/11207
    JVM_OPTS: -Xmx1024m -XX:ParallelGCThreads=2 -XX:ConcGCThreads=2 -XX:ParallelGCThreads=2 -Djava.util.concurrent.ForkJoinPool.common.parallelism=2
  steps:
    - checkout
    - restore_cache:
      key: jars-{{ checksum "src/android/consumer/recharge-consumer-android/build.gradle" }}
    - run:
      name: Download Dependencies
      command: ./gradlew androidDependencies
      pwd: src/android/consumer/recharge-consumer-android
  - save_cache:
      paths:
        - ~/.gradle
      key: jars-{{ checksum "src/android/consumer/recharge-consumer-android/build.gradle" }}
  - run:
      name: Run Tests
      command: ./gradlew testDebugUnitTest --no-daemon -Pkotlin.incremental=false --max-workers=2 # --info --stacktrace
      pwd: src/android/consumer/recharge-consumer-android
  - store_artifacts:
      path: src/android/consumer/recharge-consumer-android/build/reports
      destination: reports
  - store_test_results:
      path: src/android/consumer/recharge-consumer-android/build/test-results

#16

Regarding the timeouts, I’ve seen that too: I’ve added a few extra print outs to the unit-test runs:

testOptions {  
    unitTests.all {
        maxParallelForks = 2

        testLogging {
            if (System.getenv().containsKey("CIRCLE_BUILD_NUM")) {
                events TestLogEvent.FAILED, TestLogEvent.SKIPPED, TestLogEvent.STARTED, TestLogEvent.PASSED
            } else {
                events TestLogEvent.FAILED, TestLogEvent.SKIPPED
            }

            exceptionFormat TestExceptionFormat.FULL

            showCauses true
            showExceptions true
            showStackTraces true
        }

Take a look at https://github.com/AnySoftKeyboard/AnySoftKeyboard/blob/master/app/build.gradle#L127

Regarding the memory issues, maybe there is something I forgot to mention before… This is the project I’m running with Circle CI: https://github.com/AnySoftKeyboard/AnySoftKeyboard
There might be something I did with https://github.com/AnySoftKeyboard/AnySoftKeyboard/blob/master/gradle.properties or https://github.com/AnySoftKeyboard/AnySoftKeyboard/blob/master/circle.yml
Also, note that I’m using my own Android docker image, so that also may have something to do with it. menny/android_ndk:1.7.1


#17

Thanks for all that info. Tried making my config match yours as much as possible, still getting the 137 error. I have considerably more dependencies in my app so I’m guessing the larger memory requirements are just unavoidably going to go over the limits :sob:


#18

In that case, you can change the forks count and workers to 1, this will allow you to use a larger memory limit value


#19

Tried turning down to 1 workers and using -Xmx2048m, I also tried -Xmx1900m just in case, but they both still occasionally get the same 137 error. Super-frustrating.


#20

Ya, it is. Maybe someone from CircleCI can help with this?