Split-by-timings doesn't work for cucumber ruby

ruby cucumber formatter doesn’t work for split-by-timings what am i doing wrong here

Hi @bheemreddy181!

Could you try one of the methods outlined here for outputting the test results?

https://circleci.com/docs/2.0/collect-test-data/#cucumber

The -f pretty could be causing issues here, so may want to try and remove that.

If you continue to have issues with it after testing the above, can you submit a support request and mention this post? In addition, include a link to a build where you are having this issue?

Issues with test splitting usually require a deeper investigation and testing to resolve, so moving to a support case/having an example build usually helps with that.

Thanks!

Hi @bheemreddy181 – I wanted to followup here since I did some additional testing and investigation and believe I found the root cause.

It appears the output files from Cucumber didn’t contain the filename of the test, i.e.:

<testsuite failures="0" errors="0" skipped="1" tests="1" time="0.021938" name="Is it Friday yet?">
	<testcase classname="Is it Friday yet?" name="Sunday isn't Friday" time="0.021938">
		<skipped/>
		<system-out>
			<![CDATA[ ]]>
		</system-out>
		<system-err>
		    <![CDATA[ ]]>
		</system-err>
	</testcase>
</testsuite>

When the test splitting feature runs it tries to match the filename that it is given with a filename in the previously stored test results file. However, as you can see with the above, there isn’t a file key, like file="features/is_it_friday_yet.feature" so we are unable to split tests by timing.

With the above said, a co-worker and I worked on adding filename to the output that is generated from Cucumber. You can see the changes we made on this fork:

Once I updated my gemfile to point to the above (gem 'cucumber', github: 'mkly/cucumber-ruby') and then ran bundle install and pushed up my changes I successfully got my tests to split by timings.

Before fork:

Error autodetecting timing type, falling back to weighting by name. Autodetect no matching filename or classname.  If file names are used, double check paths for absolute vs relative.
Example input file: "features/is_it_friday_yet.feature"
Example file from timings: ""
features/is_it_friday_yet.feature

After fork:

Autodetected filename timings.
features/is_it_friday_yet.feature

The above fork is just an example so I would recommend making your own fork to maintain with the changes if you want to utilize the test by timings for Cucumber.

Hope the above helps!

2 Likes

Cool , Can we create the same PR against the actual version of cucumber ruby ? so that if it gets merged every one else can use this ?

Appreciate your investigation on this.

@bheemreddy181 You’re welcome!

I think in the current state that fork wouldn’t be able to be merged into the main cucumber-ruby, mainly since it changes the XML by default. However, you may be able to get the changes merged in if you adjusted the changes to be behind a flag, like passing --include-filename when running the tests, i.e.:

bundle exec cucumber --format junit --out /tmp/test-results/tests.xml --include-filename

That way the filename would only be included if the option was passed and won’t change the core functionality.

Hope that helps clarify!

1 Like

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.