Tests pass locally but fail on CI

Build Environment
1

Not really sure whats happening, but when I run circleci build locally all my tests pass completely fine as expected but the moment they run on Circle CI’s env they all fail and nginx seems to segfault without any error.

My builds are public and the latest can by found here https://circleci.com/gh/jloh/geojs/106

I’ve got no idea whats happening, when I try a debug via SSH I can’t find any clues apart from Nginx (Openresty technically) just fails to exists the moment it hits a requests.

Edit: Nginx/Openresty is segfaulting, but no clue as to why:

root@e50de6c934c4:~/project# nginx -p /root/project/t/servroot/ -c /root/project/t/servroot/conf/nginx.conf -g 'daemon off;'
Segmentation fault
root@e50de6c934c4:~/project#

Edit 2: Looks like its failing due to it not finding a library (which I know is installed):

root@adf23e38413f:~/project# gdb $(which nginx) core.1911.\!usr\!local\!openresty\!nginx\!sbin\!nginx
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/openresty/nginx/sbin/nginx...(no debugging symbols found)...done.
[New LWP 1911]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `nginx -p /root/project/t/servroot/ -c /root/project/t/servroot/conf/nginx.conf'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  MMDB_free_entry_data_list (entry_data_list=0x43) at maxminddb.c:1860
1860	maxminddb.c: No such file or directory.
(gdb)
2

What’s the base OS for this image? I wonder if it is something like Alpine (or other BusyBox variant) and it might be better to run on a full-fat Debian/Ubuntu derivative.

You could try compiling it remotely too - that would hopefully produce a binary that would work.

3

The image is over here, it’s based off Ubuntu already sadly. Running all the same code on a full Ubuntu VM passes fine and running the same container also passes fine.

4

Hmm, odd. A few things to try:

  • Check dmesg and other syslogs to see if there is an out of memory situation or similar
  • Check NginX logs
  • Use a more recent version of Nginx/Openresty
  • Did you try compiling Nginx/Openresty? I don’t know if that is possible.
5

Nothing in dmsg or anything. Had to get nginx to leave it’s core dumps to even get this far. Also nothing in nginx logs, had to run it in the foreground to see it was segfaulting.

AFAIK this already is the latest version of nginx/openresty. Haven’t tried compiling it manually. Not sure how that’d help given tests pass in the same container locally.

6

I’m not certain either, but it’s probably worth a go in absence of other options. A successful compilation on a machine makes it more likely to work on that machine, since by definition the linker can find all the external libraries it needs.

7

Turns out this was an issue caused by luacov. Why it was segfaulting I’m not sure and still haven’t found out. Disabling it causes the tests to run fine however. The reason they were ‘passing’ locally is because I didn’t run them locally since they don’t have the secret key :man_facepalming:

1 Like
8

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.