Test splitting by file ignoring testsuite times

Feedback & Bug Reports
1

Hi! We’re running into a problem when trying to split our tests by timings. We’re uploading our junit xml at the end of our tests, and they look like this:

<testsuites name="Mocha Tests">
  <testsuite name="First test" tests="1" errors="0" failures="0" skipped="0" timestamp="2018-02-22T17:52:22" time="5.659" file="/code/a.js">
    <testcase classname="a1" name="a1" time="2.02"></testcase>
  </testsuite>
  <testsuite name="Second test" tests="2" errors="0" failures="0" skipped="0" timestamp="2018-02-22T17:52:28" time="31.713" file="/code/b.js">
    <testcase classname="b1" name="b1" time="3.91"></testcase>
    <testcase classname="b2" name="b2" time="7.34"></testcase>
  </testsuite>
</testsuites>

Notice that our testsuite times are much higher in some cases than the sum of the testcase times. This is because setup/teardown is not included in the testcase time, but is obviously in the testsuite time.

It seems that when splitting tests by timing, circle is only attributing the testcase time to a given file, and not the testsuite time.

        {
            "classname": "a1",
            "file": "/code/a.js",
            "message": null,
            "name": "a1",
            "result": "success",
            "run_time": 2.02,
            "source": "integration",
            "source_type": "integration"
        },
        {
            "classname": "b1",
            "file": "/code/b.js",
            "message": null,
            "name": "b1",
            "result": "success",
            "run_time": 3.91,
            "source": "integration",
            "source_type": "integration"
        },
        {
            "classname": "b2",
            "file": "/code/b.js",
            "message": null,
            "name": "b2",
            "result": "success",
            "run_time": 7.34,
            "source": "integration",
            "source_type": "integration"
        },

This behavior is surprising, and it is resulting in very imbalanced test runs when we split by timings/filename, because files with a lot of setup/teardown are assumed to execute much less quickly than they actually do.

When attributing execution time to files, can we take the max(suiteTime, sum(testTime)) instead of just the sum of test times?

1 Like