Upgrading nodejs, service won't start unless I get in via SSH

Build Environment
,
1

We have been using circle.ci since a few weeks, running unit tests against a nodejs server (requiring node 0.10.40).
So far, we have our cricle.yaml running some make run, which starts a few server processes. Then, circle runs make tests, connecting to our servers, sending API calls, … testing.

As of last week, a colleague of mine started working on updating our code to work with node 4.2.
Updating our circle.yaml, I added a dependencies section, installing gcc & g++4.9 and running some update-alternatives (tip found in some other discussion, otherwise some npm modules refuse to build).
I had to make do with node 4.1 (no 4.2 available yet on circle?).
I also had to run npm install in the circle.yaml - which was done automatically before. I suppose this could be related to me overriding the dependencies section?

And my main problem: our server processes don’t start any more.
Right now, if feels like everything should work: running my tests “with ssh”, I can connect and start our servers with the command and env vars provided in our circle.yml.
However if I don’t connect and start our server processes myself, they don’t get started at all.

I patched my makefile to send both stdout & stderr to some file, when starting a server process. Usually, we can find errors such as a missing env var, missing node module, … Though when circle fails starting these processes, the corresponding log files are empty.

I have no idea why our server processes refuse to start.
Having confirmed our processes haven’t started, I can log back in via ssh, and issue a make run. Everything’s back on track.
How is it my make run could fail?

FYI, our circle.yaml:

general:
  branches:
    only:
      - staging-with-new-node
machine:
  environment:
    ENV0: 0
    ENV1: 1
  hosts:
    peeriodev.local: 127.0.0.1
  node:
    version: 4.1.0
  services:
    - redis
# circleCI required since node4.x
dependencies:
  override:
    - sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test && sudo apt-get update
    - sudo apt-get install -y gcc-4.9 g++-4.9
    - sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.6 10
    - sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-4.6 10
    - sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.9 20
    - sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-4.9 20
# //circleCI
test:
  pre:
# circleCI provides with riak-1.4.8
    - wget http://s3.amazonaws.com/downloads.basho.com/riak/2.1/2.1.1/ubuntu/trusty/riak_2.1.1-1_amd64.deb
    - sudo dpkg -i riak_2.1.1-1_amd64.deb
    - sudo sed -i 's|storage_backend.*|storage_backend = leveldb|' /etc/riak/riak.conf
    - sudo /etc/init.d/riak restart
    - sudo make inittypes
# //circleCI
# circleCI required since node4.x
    - npm install
# //circleCI
    - make run
# wait a little bit, making sure everything's started
    - sleep 10

Any advice would be welcome.

Thanks !

2

That is exactly right—overriding the dependencies section prevents the default dependency installation command (npm install in this case) from running.

Are you observing any error messages when trying to start the services? What happens if you SSH into the build and start the services manually?

3

Thanks for your answer. I will move my npm install in the dependencies block then.

Regarding my services not starting, there is no output at all, running from circle.yml.
I tried to redirect stdout and stderr from our yaml.
Failing, I moved my redirections to our Makefile.
Now I can see my log files being created. Though when I connect via SSH, they are empty.
The processes are not in my ps, already dead. I have no idea what they were saying, when failing to start.

When I connect via SSH and issue a make run, without loading any more variable that’s already been fed to circle.yml, everything works as expected: my processes start, I’m starting to see debugs in my logs file. Pretty much what I would have expected, adding my make run to my yaml.
If I connect soon enough, I can start my processes after the npm install step and before the make test step, which “fixes” my tests, …

FYI, our make run:

PORTS = 3300 3301 ...
[...]
run:
	if ! test -d node_modules; then npm install; fi; \
	mkdir -p shared/tmp/files shared/logs; \
	for port in $(PORTS); do \
	    ( PORT=$$port node foreground/app.js >/tmp/app$$port.log 2>&1 & ); \
	done; \
	( node background/app.js >/dev/null 2>&1 & ); \
	( node schedule/app.js >/dev/null 2>&1 & )
4

This is very strange. Are you sure that when you SSH in, the make run command has already started running? Does it just sit there without outputting anything to the console?

5

I am sure the make run was started: following the build/test from circle.ci, I do see this step being executed.
Having redirected output from my workers to /tmp/app$port.log, I can also confirm all log files are created, and empty.

6

Would it be possible to add logging to your Makefile? For example, outputting strings with echo could work. We would need to know what exactly is hanging the execution of the build steps.

7

Hi,

There’s no step hanging. I’m already logging each processes output in a dedicated file - incidentally, they’re all empty, which I can’t explain/working or not, our node app is pretty verbose while starting.

I can see all tasks being executed, as described in our circle.yml. Adding echoes, the last ones:

circle-wtf2.png1254×564 44.9 KB

I can confirm my processes are supposed to be started. Starting them as background, I have no real way of knowing the value for $?. Our tests start failing when I look in my ps for a running process (new “test”, to avoid starting mocha, which uses a 65s timeout).

8

I recently updated node to 4.2.2, no change.
For the record, we’re only suffering from this problem since 4.x. The unit tests work when run on some AWS instance we host ourselves. Circle tests work, if I log in via SSH on my circle instance and run my make run during thesleep 10 step. And these test also work on Circle running node 0.10.40:

circle-wtf3.png1268×574 38.6 KB

9

Sorry for multi-post: I can’t send more than one image at once, it seems, … Anyway…

Quick remark, about moving npm install in the end of my dependencies section: when I do that, I don’t see the test tasks being executed anymore. We go straight from “Dependencies” to “Database” then “Teardown”, without running anything from test (either my custom pre, or the default make test).

circle-wtf1.png1282×495 34.2 KB