The first thing I’d do is to examine your tests for memory consumption. Do they run OK locally? You could add a shutdown handler in your tests (e.g. in your bootstrap) to print the max memory that you’re using. I would expect the machine you’re using to have 4G of RAM, so if you’re getting anything over 3G (which would be highly excessive anyway) then that would need to be cut down.
Secondly, you could use the SSH feature in CircleCI to run some experiments. How far do your tests go before crashing? Does it run any at all? Is it a particular test it gets stuck on? Can you use --filter to run some tests OK?
Finally, if that does not get you anywhere, I would try bumping up to CircleCI 2.0. You get more control over your machine if you move to Docker.