Split with "tests run" not functioning

Build Environment Running Tests
,
1

Hello,
I’m using circleci tests run with more than 1 parallelism & split, however the split never happens as all tests run on each container. This is with TestCafe.
What’s missing here?

The job on circleci yml:

  release-sanity:
    circleci_ip_ranges: true
    executor: docker-executor
    parallelism: 10
    resource_class: large
    steps:
      - default_setup
      - run:
          no_output_timeout: 10m
          command: |
            circleci tests glob "tests/**/*.js" | circleci tests run --command="xargs
            npx npm run release-sanity" --split-by=filesize --verbose
      - test_reporting

The test_reporting command:

  test_reporting:
    steps:
      - store_test_results:
          path: /tmp/test-results
      - store_artifacts: 
          path: artifacts/screenshots
      - store_artifacts:
          path: artifacts/recordings

The script in package.json:
"release-sanity": "ENV=release SCOPE=sanity testcafe -r spec,xunit:/tmp/test-results/testcafe/results.xml"

2

You are currently executing the following command in each instance

npx npm run release-sanity

what does the file release-sanity contain?

3

My understanding of this:

circleci tests glob "tests/**/*.js" | circleci tests run --command="xargs
            npx npm run release-sanity" --split-by=filesize --verbose

is that all tests are grabbed, the split applies on them to determine which tests should get triggered on each container, and then it triggers “release-sanity” script.
The script will filter testcafe tests based on the defined environment & meta scope and run them accordingly.

But this obviously is not happening.

4

Yes, that is how it should be working and what you have shown matches all the examples in the docs, but I can not currently comment on what is happening within the release-sanity script.

5

Also as you have included --verbose on the command line can you please post the resulting output. This will show up in the step output as a number of line in the format of

DEBU[date/time] text

6

Here’s the workflow output.
I have parallelism of 2. Total of 4 sanity tests.
There are 2 test files, courses_page_tests_1.js has 3 sanity tests & login_page_tests.js has one.

7

So the CircleCI is

  • correctly creating the 2 independent environments to run the tests in.
  • splitting the 4 tests across the 2 environments. You can see this in the log as
    the 4 tests are spread across the 2 DEBU Received lines.

I asked about the sanity_run script because this is what is run in each container with the test script names being passed to it as parameters. So what is the script doing with the detail? From how you describe the issue and the few lines of output on these screenshots, I wonder if the script is processing the list passed to it.

8

The sanity_run script is supposed to:

  • Check if the env & scope are passed in the script, in our case they are.
    then it
  • Filter the list of tests (passed through Circle) & only run the ones with “sanity” scope meta. The scope meta is defined on the test itself.
    This happens by defining the filter in testcafe config file.

What I don’t really understand is that if each container receives the correct file, then the script should filter those passed tests in that file only & run the ones with sanity scope, instead of executing all sanity scope tests in all files.

I.E container 0 receives the courses_page_tests file. Then it should run the 3 sanity tests inside this file & keep the other sanity test on login_page file to run on container 1.

9

Well you can check what is being passed to your script by changing

circleci tests glob "tests/**/*.js" | circleci tests run --command="xargs
            npx npm run release-sanity" --split-by=filesize --verbose

to

circleci tests glob "tests/**/*.js" | circleci tests run --command="xargs
            echo" --split-by=filesize --verbose

This should display the list of tests that should be run within the container as they are just appended to the command line by the xargs command.

I’m using the following simplified config.yml to test things out

version: 2.1

jobs:
    build-and-test:
        docker:
          - image: cimg/node:16.10
        resource_class: large
        parallelism: 2
        steps:
            - run: 
                command: |
                    mkdir tests
                    echo "echo test 1 $1 $2" > tests/test1.sh
                    echo "echo test 2 $1 $2" > tests/test2.sh
                    echo "echo test 3 $1 $2" > tests/test3.sh
                    echo "echo test 4 $1 $2" > tests/test4.sh
                    chmod +x tests/*.sh
            - run:
                name: Test application
                command: |
                    TEST=$(circleci tests glob "tests/*.sh")
                    echo $TEST
                    echo "$TEST" | circleci tests run --command="xargs echo " --split-by=name --timings-type=name --verbose

workflows:
    sample:
      jobs:
        - build-and-test

This outputs the following to container 0, which is what I would expect.

tests/test1.sh tests/test2.sh tests/test3.sh tests/test4.sh
Installing circleci-tests-plugin-cli plugin.
circleci-tests-plugin-cli plugin Installed. Version: 1.0.7332-dfcf8a1
DEBU[2024-02-23T09:38:25Z] Attempting to read from stdin. This will hang if no input is provided. 
DEBU[2024-02-23T09:38:25Z] starting execution                           
DEBU[2024-02-23T09:38:25Z] Received: tests/test1.sh tests/test3.sh      
tests/test1.sh tests/test3.sh
DEBU[2024-02-23T09:38:25Z] Completed tests                              
DEBU[2024-02-23T09:38:25Z] ending execution                             

CircleCI received exit code 0
10

Thanks for providing your example!
With echo, it lists the files. Container 0 received courses_page file which is expected.
Still not sure why each container does not run the tests on the received file only.

11

OK, from that output it is clear that the CircleCI environment is managing to correctly split out the test files found in the directory, but in many ways that is the easy bit.

You will need to take a look at how the tests and release-sanity script are written as they are the only place where a rescanning of the test directory can take place to cause all the tests to run within a single container.