Getting "No timing found" errors using test splits by timing

Build Environment Running Tests
1

we’re running 20864 tests in our pipeline using timing-based test splits (circleci tests split --split-by=timings).

We keep seeing “No timing found for X” messages for some of our tests, despite confirming that they are running and the test results with timing are included in the test XML. You can also see under the TESTS tab for out workflow that all 20864 tests are running.

Looking at the results.json file confirms that some of our tests are not being recorded there. I counted only 20812 tests reported in the file which would explain the “No timing found” errors.

Are we hitting some kind of undocumented limitation? The file size of this JSON is in fact pretty large at 6MB.

1 Like
2

@jasonwhipp Could you share the relevant portion of your config? Are you saving the JUINIT XML format test results to the save_test_results step?

3

Yes we are, when we execute the tests we use

–format RspecJunitFormatter
–out /tmp/test-results/rspec.xml

store_test_results:
path: /tmp/test-results

4

What does the output of the save test results step look like? Has it been successfully saved?
Provide as much information as possible about your current situation. Screenshots and code will help

The results json file you mentioned, is that saved to an artifact? How does it compare to the XML output?

5

We logged into our test container and SCP’d the “circle-test-results/results.json” file back for analysis. This results.json file has 20812 entries.

Circleci itself reports that 20864 tests ran.

For each test that reported “No timing found for”, we noticed that it is missing from results.json

No tests are missing from the test XML and circleci reports all of the tests as running successfully.

Therefor we concluded circle-test-results/results.json is missing 52 tests, but we don’t know why.

1 Like
6

Screenshot 2023-10-13 at 5.21.52 PM
Screenshot 2023-10-13 at 5.21.52 PM2016×1430 459 KB

    - run:                                                                                                                                                         
        name: RSpec                                                                                                                                                
        no_output_timeout: 30m                                                                                                                                     
        command: |                                                                                                                                                 
          echo "export RP_FAILURE_TYPE='outside of run_rspec'" >> $BASH_ENV                                                                                        
          touch log/bullet.log log/rspec.log log/tests.txt                                                                                                         
          TEST_FILES=$(circleci tests glob "spec/**/*_spec.rb")                                                                                                    
          echo "$TEST_FILES" | circleci tests run --command="xargs timeout 1h .circleci/run_rspec" --verbose --split-by=timings                                    
          mkdir rspec_logs bullet_logs                                                                                                                             
          cp log/tests.txt rspec_logs/tests_${CIRCLE_NODE_INDEX}.txt                                                                                               
          cp log/rspec.log rspec_logs/rspec_${CIRCLE_NODE_INDEX}.log                                                                                               
          [ ! -f log/only_failures_tests.txt ] || cp log/only_failures_tests.txt rspec_logs/only_failures_tests_${CIRCLE_NODE_INDEX}.txt                           
          cp log/bullet.log bullet_logs/bullet_${CIRCLE_NODE_INDEX}.log                                                                                            
          if [ -s "coverage/.resultset.json" ]                                                                                                                     
          then                                                                                                                                                     
            mkdir coverage_files                                                                                                                                   
            cp coverage/.resultset.json coverage_files/${CIRCLE_NODE_INDEX}.resultset.json || echo -n ""                                                           
          fi                                                                                                                                                       
                                                                                                                                                                   
    - slack/notify:                                                                                                                                                
        event: fail                                                                                                                                                
        custom: |                                                                                                                                                  
          {                                                                                                                                                        
            "text": ":x: RSpecs FAILED.",                                                                                                                          
            "blocks": [                                                                                                                                            
              {                                                                                                                                                    
                "type": "section",                                                                                                                                 
                "text": {                                                                                                                                          
                  "type": "mrkdwn",                                                                                                                                
                  "text": ":x: <$CIRCLE_BUILD_URL|RSpecs FAILED. $SLACK_PARAM_MENTIONS $CIRCLE_JOB $CIRCLE_NODE_INDEX ${CIRCLE_BRANCH:-$CIRCLE_TAG}> $RP_FAILURE_T\
YPE"                                                                                                                                                               
                }                                                                                                                                                  
              }                                                                                                                                                    
            ]                                                                                                                                                      
          }                                                                                                                                                        
                                                                                                                                                                   
    - persist_to_workspace:                                                                                                                                        
        root: .                                                                                                                                                    
        paths:                                                                                                                                                     
          - rspec_logs                                                                                                                                             
          - coverage_files                                                                                                                                         
          - bullet_logs                                                                                                                                            
                                                                                                                                                                   
    - store_test_results:                                                                                                                                          
        path: log
7

Trying to investigate if the issue is that the rspec junit files are listed with ./ at the begining of the file path, but the command parameters do not have the ./ at the beginning. It seems that circleci tests glob "./spec/**/*_spec.rb" doesn’t seem to add it either. It’s just bad that the timing information is not found and it is in the xml file. This makes our run take a few extra minutes.