There’s not much to critique in your config file, so I’d suggest debugging on the server.
You can get an SSH session on a failing build, and run the tests from the console. I’d suggest using Nano or some other console editor to make changes to your tests (e.g. add debug statements) to see what is going on.
Check that you are waiting for your server to become available before using it - you may have a race condition if you are starting it in the background. Or you might be listening on the wrong IP/interface (though I’d doubt this if they sometimes work).
I would assume you’d need to modify the file and recompile it in the console for Go.