Selenium fails only when run in parallel tests

I am using protractor/selenium to run some tests in Chrome. When I use a single container everything works as expected. When I introduce parallel tests, it works fine on the first container, but fails on the second container telling me: chrome not reachable.

I SSH’d into the failing container and found that trying to run google-chrome with xvfb-run resulted in an error and I could only get this to work by using a non-default display for xvfb, such as :100 instead. However, this was also true for the working container.

Why would might parallel tests cause this issue? Are the two containers not completely separate?
Has anyone got selenium tests running in parallel containers?