Django: Problems with test splitting and overall performance

Build Environment
, , , ,
1

Hey :slight_smile: Our CircleCI takes a bit too long for our taste and makes iterating very difficult.

I am trying to set up code splititng our in our config but it seems like the command circleci tests glob "/**/tests.py" takes like forever to run… We have around 1086 tests. This number comes from Djangos test runner.

The point of the splitting is of course to make it faster on CI, but the provided command hangs for 10m and then exits… not sure what i do wrong?

Our monorepo consits of a lot of small tests.py files, but also nested folder structure: such as tests folders with integration and unit test folders, in which test_ files sit.

Please take a look at the config and suggest big mistakes we might make.

version: 2.1

orbs:
  node: circleci/node@5.0.0
  python: circleci/python@2.0.1
  browser-tools: circleci/browser-tools@1.2.4

workflows:
  version: 2
  build:
    jobs:
      - build_frontend:
          filters:
            branches:
              ignore: /^master$/
      - build_backend:
          filters:
            branches:
              ignore: /^master$/
      - run_tests:
          filters:
            branches:
              ignore: /^master$/
          requires:
            - build_backend
            - build_frontend

executors:
  backend:
    docker:
    - image: cimg/python:3.9.9-browsers
      environment:
        DATABASE_URL: postgres://postgres:postgres@localhost/postgres?sslmode=disable
        PARALLEL_BUILD_FRONTEND: 1
    - image: postgres:latest
      environment:
        POSTGRES_HOST_AUTH_METHOD: trust

jobs:
  build_frontend:
    executor: node/default
    resource_class: large
    steps:
      - checkout
      - run: git log --format="%H" -n1 web_ui/frontend > frontend-git-revision.txt && cat frontend-git-revision.txt
      - restore_cache:
          key: frontendbuild-{{ checksum "frontend-git-revision.txt" }}
      - restore_cache:
          key: v1-npm-deps-{{ checksum "web_ui/frontend/package-lock.json" }}
      - run:
          name: "Build frontend"
          command: |
            ./scripts/build_frontend.sh
      - run:
          name: "Extract frontend translations"
          command: |
            npm run --prefix web_ui/frontend extract-translations > /dev/null
      - save_cache:
          key: frontendbuild-{{ checksum "frontend-git-revision.txt" }}
          paths:
            - "web_ui/frontend"
            - "web_ui/static/dist"
            - "web_ui/frontend/locale/messages.pot"
      - save_cache:
          key: v1-npm-deps-{{ checksum "web_ui/frontend/package-lock.json" }}
          paths:
            - "web_ui/frontend/node_modules"
      - persist_to_workspace:
          root: .
          paths:
            - locale
            - web_ui/frontend
            - web_ui/static/dist
            - web_ui/frontend/node_modules
  build_backend:
    executor: backend
    resource_class: xlarge
    steps:
      - checkout
      - run:
          name: "Install system dependencies"
          command: |
            sudo apt-get update && sudo apt-get install -y gettext
      - python/install-packages:
          pkg-manager: pip
          pip-dependency-file: requirements-dev.txt
          pypi-cache: true
          cache-version: venv-v1.0-{{ checksum "requirements-dev.txt" }}
      - run:
          name: "Check that no model changes are missing migrations"
          command: |
            ./manage.py makemigrations --check
  make_test_files:
    executor: backend
    resource_class: xlarge
    steps:
      - checkout
      - python/install-packages:
          pkg-manager: pip
          pip-dependency-file: requirements-dev.txt
          pypi-cache: true
          cache-version: venv-v1.0-{{ checksum "requirements-dev.txt" }}'
      - run:
          command: |
            set -e
            # get all test files
            TESTFILES=$(circleci tests glob "/**/tests.py")
            echo $TESTFILES | tr ' ' '\n' | sort | uniq > circleci_test_files.txt
            cat circleci_test_files.txt
            TESTFILES=$(circleci tests split --split-by=timings circleci_test_files.txt)
            # massage filepaths into format manage.py test accepts
            TESTFILES=$(echo $TESTFILES | tr "/" "." | sed 's/.py//g')
            echo $TESTFILES
            ./manage.py test --keepdb --parallel 8 --exclude-tag=functional --timing
  run_tests:
    executor: backend
    parallelism: 5
    resource_class: xlarge
    steps:
      - checkout
      - attach_workspace:
          at: .
      - browser-tools/install-chrome
      - browser-tools/install-chromedriver
      - python/install-packages:
          pkg-manager: pip
          pip-dependency-file: requirements-dev.txt
          pypi-cache: true
          cache-version: venv-v1.0-{{ checksum "requirements-dev.txt" }}
      - run:
          name: "Backend tests"
          command: |
            ./manage.py test --keepdb --parallel 10 --exclude-tag=functional --timing
      - run: git log --format="%H" -n1 web_ui/frontend > frontend-git-revision.txt && cat frontend-git-revision.txt
      - restore_cache:
          key: frontendbuild-{{ checksum "frontend-git-revision.txt" }}
      - run:
          name: "Functional tests"
          command: |
            ./manage.py test --tag functional
      - store_artifacts:
          path: /tmp/artifacts
      - store_artifacts:
          path: test-results
      - store_test_results:
          path: test-results

The way our setup works is:

  • build frontend first
  • build backend
  • attach both to a workspace where we run functional and unit tests

As mentioned, we would like to split up our unit tests, but the provided config in the docs does not work, as the glob never finishes, even if we narrow down the pattern.

Many thanks .)

1 Like
2

Facing the same issue, any solutions now? Thanks.

3

Below is my workaround (with jest) to make tests split work as expected. The other languages/framework should be similar.

As circleci tests glob hangs for unacceptable duration, I wrote a custom script to get relevant tests and pass to circleci tests split.

The rough idea is as below:

// listTests.ts
import { execSync } from 'node:child_process'

const testFiles = execSync(`yarn test --listTests`)
  .toString()
  .split('\n')
  // they are absolute paths, and  you might want to add custom regex to extract filepath to be tally with junit xml
  .map(filepath => filepath.match(RegExp(`.*some-unwanted-local-path/(.*)`))?.[1])
  .filter(Boolean)
  .join('\n')

const filePath = join(__dirname, `../generated.testFiles`)
writeFileSync(filePath, testFiles)

In package.json

// package.json
"test:listTests" : "ts-node path-to/listTests.ts"

And in CircleCI config:

// config.yml
...
      - run:
          name: Extract testFiles
          command: |
            yarn test:listTests
            cat generated.testFiles
      - run:
          name: Run tests
          command: |
            TEST=$(cat generated.testFiles | circleci tests split --split-by=timings)
            yarn test $TEST
      - store_test_results:
          path: ./junit
      - store_artifacts:
          path: ./junit