Circle CI v2 and Android - Memory issues?

I recommend holding off on Android until we publish official docs. It’s really not a fun process and we still don’t have a green build that we can show off. We have a lot of hours into it and I can’t in good conscience advise you to squander that time.

1 Like

@rohara - Following our conversations and workarounds of last week I’ve just managed to get my first green build of my Android app on CircleCI 2.0!

Want to show that one off?

:blush:

Here’s my circle.yml

version: 2
jobs:
  build:
    docker:
      - image: bitriseio/docker-android
    working_directory: ~/OceanLife
    steps:
      - checkout
      - run:
          name: Assemble & Test (Amazon Flavour)
          command: ./gradlew assembleWithAmazon testWithAmazonDebugUnitTest
          environment:
            TERM: dumb
      - run:
          name: Assemble & Test (Google Flavour)
          command: ./gradlew assembleWithGoogle testWithGoogleDebugUnitTest
          environment:
            TERM: dumb

Proof!

1 Like

@BrantApps I am DEFINITELY showing this off internally.

1 Like

I got a green build with Docker as well:
https://circleci.com/gh/AnySoftKeyboard/AnySoftKeyboard/330

The issue was Docker terminating Gradle executors because they consume too much memory.
This is because Java runtime is unaware of cgroups limitations, so I had to be explicit about the memory limitations by adding the environment variable _JAVA_OPTIONS: "-Xmx1024m"

@rohara, what is the Docker memory quota? And can we have that as an environment variable (as well as the CPU count).

1 Like

I will open a feature request. Two cores, four gigabytes of RAM.

The most I could get was: _JAVA_OPTIONS: "-Xmx1500m -XX:ParallelGCThreads=2 -XX:ConcGCThreads=2 -XX:ParallelGCThreads=2 -Djava.util.concurrent.ForkJoinPool.common.parallelism=2"
But that might be because I allow two instances of the JVM.

Anyhow, thanks a lot for the help. I’m happy with the speed boost.

I’m going to add deploy next.

1 Like

Any updated on this topic?
We have run into the same issue. Unless we set the heap to something lower than 1/4 of the container memory (1024m in this case), our java processes are killed with a 137 exit code. So at the moment we are using _JAVA_OPTIONS: "-Xmx1024m" but this is really not enough for Gradle when building a large Android project. Given that the Circle 2 containers are 4Gb, we should be able to use a heap much larger than 1Gb.

1 Like

I’m not sure if this is relevant to this specific case, but I was having a similar issue (my build is not Android, but is running Gradle in CircleCI v2 Docker container; the process is being killed with exit code 137). Limiting memory did not solve the problem in my case.

I discovered that Gradle was running with 32 workers, which by far exceeds the 2 CPUs assigned to the container. Adding the --info flag to the end of the Gradle command confirms this. In my case, I changed the following command:
./gradlew test --no-daemon
to
./gradlew test --no-daemon --max-workers 2

Hopefully this information helps someone if they are encountering similar issues.

Yes, that was also required for me. I passed the Max threads count as a JVM environment variable and also set it in the gradle.properties file

I’ve tried all the suggestions from this thread but I still see the occasional Process ‘Gradle Test Executor 1’ finished with non-zero exit value 137 error, and I see more frequent Too long with no output (exceeded 10m0s) errors that I can’t figure out, but I assume something is hanging/deadlocking because of the memory restrictions. The only suggestion from this thread I haven’t tried is setting the max threads in gradle,properties, bc I couldn’t figure out what key to use. Menny, can you post that?

FWIW, here’s my circle config snippet for my app (i’m in a mono-repo, hence the pwd noise):

android-consumer:
  working_directory: ~/recharge
  docker:
    - image: circleci/android:api-26-alpha
  environment:
    # from https://discuss.circleci.com/t/circle-ci-v2-and-android-memory-issues/11207
    JVM_OPTS: -Xmx1024m -XX:ParallelGCThreads=2 -XX:ConcGCThreads=2 -XX:ParallelGCThreads=2 -Djava.util.concurrent.ForkJoinPool.common.parallelism=2
  steps:
    - checkout
    - restore_cache:
      key: jars-{{ checksum "src/android/consumer/recharge-consumer-android/build.gradle" }}
    - run:
      name: Download Dependencies
      command: ./gradlew androidDependencies
      pwd: src/android/consumer/recharge-consumer-android
  - save_cache:
      paths:
        - ~/.gradle
      key: jars-{{ checksum "src/android/consumer/recharge-consumer-android/build.gradle" }}
  - run:
      name: Run Tests
      command: ./gradlew testDebugUnitTest --no-daemon -Pkotlin.incremental=false --max-workers=2 # --info --stacktrace
      pwd: src/android/consumer/recharge-consumer-android
  - store_artifacts:
      path: src/android/consumer/recharge-consumer-android/build/reports
      destination: reports
  - store_test_results:
      path: src/android/consumer/recharge-consumer-android/build/test-results
1 Like

Regarding the timeouts, I’ve seen that too: I’ve added a few extra print outs to the unit-test runs:

testOptions {  
    unitTests.all {
        maxParallelForks = 2

        testLogging {
            if (System.getenv().containsKey("CIRCLE_BUILD_NUM")) {
                events TestLogEvent.FAILED, TestLogEvent.SKIPPED, TestLogEvent.STARTED, TestLogEvent.PASSED
            } else {
                events TestLogEvent.FAILED, TestLogEvent.SKIPPED
            }

            exceptionFormat TestExceptionFormat.FULL

            showCauses true
            showExceptions true
            showStackTraces true
        }

Take a look at https://github.com/AnySoftKeyboard/AnySoftKeyboard/blob/master/app/build.gradle#L127

Regarding the memory issues, maybe there is something I forgot to mention before… This is the project I’m running with Circle CI: https://github.com/AnySoftKeyboard/AnySoftKeyboard
There might be something I did with https://github.com/AnySoftKeyboard/AnySoftKeyboard/blob/master/gradle.properties or https://github.com/AnySoftKeyboard/AnySoftKeyboard/blob/master/circle.yml
Also, note that I’m using my own Android docker image, so that also may have something to do with it. menny/android_ndk:1.7.1

Thanks for all that info. Tried making my config match yours as much as possible, still getting the 137 error. I have considerably more dependencies in my app so I’m guessing the larger memory requirements are just unavoidably going to go over the limits :sob:

1 Like

In that case, you can change the forks count and workers to 1, this will allow you to use a larger memory limit value

Tried turning down to 1 workers and using -Xmx2048m, I also tried -Xmx1900m just in case, but they both still occasionally get the same 137 error. Super-frustrating.

1 Like

Ya, it is. Maybe someone from CircleCI can help with this?

I have no idea why, but adding “resource_class: large” to my circle config yaml seems to have fixed the issue.

Wow… that’s amazing.
https://circleci.com/docs/2.0/configuration-reference/#resource_class

To solve all future/remaining doubts: https://circleci.com/blog/how-to-handle-java-oom-errors/

2 Likes

To anyone still having problems (like me) note that Gradle ignores java opts and other parameters when it runs unit tests. I put an Xmx in this spot and I was golden - https://stackoverflow.com/questions/39437572/gradle-equivalent-of-test-configuration-block-for-android