We’re using circle to build, test, and deploy a go webapp using docker. We also use mongodb as our database and we have unit tests that interact with mongodb.
On local the command “go test ./…” passes fine, and until recently, they also passed in circleci. Recently, however, some of the tests have been failing only in circleci, and the failures are inconsistent. Different tests fail across rebuilds, and the errors are related to documents not being found in mongo even after data has been seeded.
For example:
The first rectangle is showing a readout of the seed data needed from mongo for the test, and correctly shows the values that were seeded. Later in the test, we do another readout from mongo and the document is empty, which leads to the error highlighted in the 3rd rectangle where mongo is returning a “not found” error.
None of the code between these lines does any remove or update operations.
Multiple tests error as a result of this kind of inconsistency, where data is being written into mongo, but later when it’s read out is no longer there.
We tried setting some writeConcern values for the mongo session (FSync: true, J:true, W:1) thinking that maybe it was the writes were not getting through in time for the reads, but they did not have any effect. We also ensured that the same session connection was being used for the duration of each test so that writes and reads were being done through the same connection as opposed to a separate connection.
One interesting observation is that the test failures only seem to happen when using the command “go test ./…” If instead we run subsequent go test commands for each package (e.g. "go test ./package1 && go test ./package2) then the tests seem to run fine, so I’m wondering if there’s some sort of concurrency issue going on when using “./…” to run all the tests in a directory/subdirectories.
Any thoughts and ideas are appreciated, thanks!