# Split with "tests run" not functioning

**URL:** <https://discuss.circleci.com/t/split-with-tests-run-not-functioning/50521>\
**Category:** Running Tests\
**Tags:** parallelism, circle-yml\
**Created:** [February 20, 2024, 8:10pm UTC](https://discuss.circleci.com/t/split-with-tests-run-not-functioning/50521 "2024-02-20T20:10:00Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![RA\_ci](https://avatars.discourse-cdn.com/v4/letter/r/bc8723/32.png) [@RA\_ci](https://discuss.circleci.com/u/RA_ci)\
**Post date:** [February 20, 2024, 8:10pm UTC](https://discuss.circleci.com/t/split-with-tests-run-not-functioning/50521/1 "2024-02-20T20:10:00Z")

</div>

Hello,  
I’m using `circleci tests run` with more than 1 `parallelism` & split, however the split never happens as all tests run on each container. This is with TestCafe.  
What’s missing here?

The job on circleci yml:

```auto
  release-sanity:
    circleci_ip_ranges: true
    executor: docker-executor
    parallelism: 10
    resource_class: large
    steps:
      - default_setup
      - run:
          no_output_timeout: 10m
          command: |
            circleci tests glob "tests/**/*.js" | circleci tests run --command="xargs
            npx npm run release-sanity" --split-by=filesize --verbose
      - test_reporting

```

The test\_reporting command:

```auto
  test_reporting:
    steps:
      - store_test_results:
          path: /tmp/test-results
      - store_artifacts: 
          path: artifacts/screenshots
      - store_artifacts:
          path: artifacts/recordings

```

The script in package.json:  
` "release-sanity": "ENV=release SCOPE=sanity testcafe -r spec,xunit:/tmp/test-results/testcafe/results.xml"`

---

<div class="post-metadata">

**Author:** ![rit1010](https://sea2.discourse-cdn.com/circleci/user_avatar/discuss.circleci.com/rit1010/32/34738_2.png) [@rit1010](https://discuss.circleci.com/u/rit1010)\
**Post date:** [February 21, 2024, 8:12pm UTC](https://discuss.circleci.com/t/split-with-tests-run-not-functioning/50521/2 "2024-02-21T20:12:57Z")

</div>

You are currently executing the following command in each instance

```auto
npx npm run release-sanity

```

what does the file release-sanity contain?

---

<div class="post-metadata">

**Author:** ![RA\_ci](https://avatars.discourse-cdn.com/v4/letter/r/bc8723/32.png) [@RA\_ci](https://discuss.circleci.com/u/RA_ci)\
**Post date:** [February 22, 2024, 3:49am UTC](https://discuss.circleci.com/t/split-with-tests-run-not-functioning/50521/3 "2024-02-22T03:49:32Z")

</div>

My understanding of this:

```auto
circleci tests glob "tests/**/*.js" | circleci tests run --command="xargs
            npx npm run release-sanity" --split-by=filesize --verbose

```

is that all tests are grabbed, the split applies on them to determine which tests should get triggered on each container, and then it triggers “release-sanity” script.  
The script will filter testcafe tests based on the defined environment & meta scope and run them accordingly.

But this obviously is not happening.

---

<div class="post-metadata">

**Author:** ![rit1010](https://sea2.discourse-cdn.com/circleci/user_avatar/discuss.circleci.com/rit1010/32/34738_2.png) [@rit1010](https://discuss.circleci.com/u/rit1010)\
**Post date:** [February 22, 2024, 9:57am UTC](https://discuss.circleci.com/t/split-with-tests-run-not-functioning/50521/4 "2024-02-22T09:57:42Z")

</div>

Yes, that is how it should be working and what you have shown matches all the examples in the docs, but I can not currently comment on what is happening within the release-sanity script.

---

<div class="post-metadata">

**Author:** ![rit1010](https://sea2.discourse-cdn.com/circleci/user_avatar/discuss.circleci.com/rit1010/32/34738_2.png) [@rit1010](https://discuss.circleci.com/u/rit1010)\
**Post date:** [February 22, 2024, 3:10pm UTC](https://discuss.circleci.com/t/split-with-tests-run-not-functioning/50521/5 "2024-02-22T15:10:31Z")

</div>

Also as you have included --verbose on the command line can you please post the resulting output. This will show up in the step output as a number of line in the format of

DEBU[date/time] text

---

<div class="post-metadata">

**Author:** ![RA\_ci](https://avatars.discourse-cdn.com/v4/letter/r/bc8723/32.png) [@RA\_ci](https://discuss.circleci.com/u/RA_ci)\
**Post date:** [February 22, 2024, 7:56pm UTC](https://discuss.circleci.com/t/split-with-tests-run-not-functioning/50521/6 "2024-02-22T19:56:05Z")

</div>

Here’s the [workflow output](https://d.pr/i/dO1Wsy).  
I have parallelism of 2. Total of 4 sanity tests.  
There are 2 test files, `courses_page_tests_1.js` has 3 sanity tests & `login_page_tests.js` has one.

---

<div class="post-metadata">

**Author:** ![rit1010](https://sea2.discourse-cdn.com/circleci/user_avatar/discuss.circleci.com/rit1010/32/34738_2.png) [@rit1010](https://discuss.circleci.com/u/rit1010)\
**Post date:** [February 23, 2024, 12:03am UTC](https://discuss.circleci.com/t/split-with-tests-run-not-functioning/50521/7 "2024-02-23T00:03:18Z")

</div>

So the CircleCI is

- correctly creating the 2 independent environments to run the tests in.
- splitting the 4 tests across the 2 environments. You can see this in the log as  
the 4 tests are spread across the 2 DEBU Received lines.

I asked about the sanity\_run script because this is what is run in each container with the test script names being passed to it as parameters. So what is the script doing with the detail? From how you describe the issue and the few lines of output on these screenshots, I wonder if the script is processing the list passed to it.

---

<div class="post-metadata">

**Author:** ![RA\_ci](https://avatars.discourse-cdn.com/v4/letter/r/bc8723/32.png) [@RA\_ci](https://discuss.circleci.com/u/RA_ci)\
**Post date:** [February 23, 2024, 3:10am UTC](https://discuss.circleci.com/t/split-with-tests-run-not-functioning/50521/8 "2024-02-23T03:10:21Z")

</div>

The `sanity_run` script is supposed to:

- Check if the env & scope are passed in the script, in our case they are.  
then it
- Filter the list of tests (passed through Circle) & only run the ones with “sanity” scope meta. The scope meta is defined on the test itself.  
This happens by defining the [filter](https://d.pr/i/FTZRE6) in testcafe config file.

What I don’t really understand is that if each container receives the correct file, then the script should filter those passed tests in that file only & run the ones with sanity scope, instead of executing all sanity scope tests in all files.

I.E container 0 receives the courses\_page\_tests file. Then it should run the 3 sanity tests inside this file & keep the other sanity test on login\_page file to run on container 1.

---

<div class="post-metadata">

**Author:** ![rit1010](https://sea2.discourse-cdn.com/circleci/user_avatar/discuss.circleci.com/rit1010/32/34738_2.png) [@rit1010](https://discuss.circleci.com/u/rit1010)\
**Post date:** [February 23, 2024, 9:44am UTC](https://discuss.circleci.com/t/split-with-tests-run-not-functioning/50521/9 "2024-02-23T09:44:59Z")

</div>

Well you can check what is being passed to your script by changing

```auto
circleci tests glob "tests/**/*.js" | circleci tests run --command="xargs
            npx npm run release-sanity" --split-by=filesize --verbose

```

to

```auto
circleci tests glob "tests/**/*.js" | circleci tests run --command="xargs
            echo" --split-by=filesize --verbose

```

This should display the list of tests that should be run within the container as they are just appended to the command line by the xargs command.

I’m using the following simplified config.yml to test things out

```auto
version: 2.1

jobs:
    build-and-test:
        docker:
          - image: cimg/node:16.10
        resource_class: large
        parallelism: 2
        steps:
            - run: 
                command: |
                    mkdir tests
                    echo "echo test 1 $1 $2" > tests/test1.sh
                    echo "echo test 2 $1 $2" > tests/test2.sh
                    echo "echo test 3 $1 $2" > tests/test3.sh
                    echo "echo test 4 $1 $2" > tests/test4.sh
                    chmod +x tests/*.sh
            - run:
                name: Test application
                command: |
                    TEST=$(circleci tests glob "tests/*.sh")
                    echo $TEST
                    echo "$TEST" | circleci tests run --command="xargs echo " --split-by=name --timings-type=name --verbose

workflows:
    sample:
      jobs:
        - build-and-test

```

This outputs the following to container 0, which is what I would expect.

```auto
tests/test1.sh tests/test2.sh tests/test3.sh tests/test4.sh
Installing circleci-tests-plugin-cli plugin.
circleci-tests-plugin-cli plugin Installed. Version: 1.0.7332-dfcf8a1
DEBU[2024-02-23T09:38:25Z] Attempting to read from stdin. This will hang if no input is provided. 
DEBU[2024-02-23T09:38:25Z] starting execution                           
DEBU[2024-02-23T09:38:25Z] Received: tests/test1.sh tests/test3.sh      
tests/test1.sh tests/test3.sh
DEBU[2024-02-23T09:38:25Z] Completed tests                              
DEBU[2024-02-23T09:38:25Z] ending execution                             

CircleCI received exit code 0

```

---

<div class="post-metadata">

**Author:** ![RA\_ci](https://avatars.discourse-cdn.com/v4/letter/r/bc8723/32.png) [@RA\_ci](https://discuss.circleci.com/u/RA_ci)\
**Post date:** [February 24, 2024, 6:02am UTC](https://discuss.circleci.com/t/split-with-tests-run-not-functioning/50521/10 "2024-02-24T06:02:08Z")

</div>

Thanks for providing your example!  
With echo, it lists the [files](https://d.pr/i/RVj1UW). Container 0 received courses\_page file which is expected.  
Still not sure why each container does not run the tests on the received file only.

---

<div class="post-metadata">

**Author:** ![rit1010](https://sea2.discourse-cdn.com/circleci/user_avatar/discuss.circleci.com/rit1010/32/34738_2.png) [@rit1010](https://discuss.circleci.com/u/rit1010)\
**Post date:** [February 24, 2024, 12:18pm UTC](https://discuss.circleci.com/t/split-with-tests-run-not-functioning/50521/11 "2024-02-24T12:18:36Z")

</div>

OK, from that output it is clear that the CircleCI environment is managing to correctly split out the test files found in the directory, but in many ways that is the easy bit.

You will need to take a look at how the tests and release-sanity script are written as they are the only place where a rescanning of the test directory can take place to cause all the tests to run within a single container.
