Better way to download, build and install dependency? Cache result?

docker

#1

I am migrating a project from building on macOS to building on Ubuntu Linux via Docker.

The project has a dependency on xar which is available by default on macOS, but not on Ubuntu.

Unfortunately there is not apt-get install xar, but there is a website:
http://mackyle.github.io/xar/

From the instructions there I created this step:

      - run:
          name: Install additional dependency xar
          command: |
            cd /tmp
            mkdir download
            cd download
            wget https://github.com/downloads/mackyle/xar/xar-1.6.1.tar.gz
            tar -xzf xar-1.6.1.tar.gz
            cd xar-1.6.1
            ./autogen.sh --noconfigure
            ./configure
            make
            sudo make install

It runs just fine and downloads, unpacks, builds and installs xar:

Summary
#!/bin/bash --login -eo pipefail
cd /tmp
mkdir download
cd download
wget https://github.com/downloads/mackyle/xar/xar-1.6.1.tar.gz
tar -xzf xar-1.6.1.tar.gz
cd xar-1.6.1
./autogen.sh --noconfigure
./configure
make
sudo make install
--2017-12-15 17:00:44--  https://github.com/downloads/mackyle/xar/xar-1.6.1.tar.gz
Resolving github.com (github.com)... 192.30.253.112, 192.30.253.113
Connecting to github.com (github.com)|192.30.253.112|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github.s3.amazonaws.com/downloads/mackyle/xar/xar-1.6.1.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAISTNZFOVBIJMK3TQ%2F20171215%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20171215T170044Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host&X-Amz-Signature=958f36d2a1244a75072bb692cdbc5e5aaff7fead0f3742a2bbfb7d409ac058d1 [following]
--2017-12-15 17:00:44--  https://github.s3.amazonaws.com/downloads/mackyle/xar/xar-1.6.1.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAISTNZFOVBIJMK3TQ%2F20171215%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20171215T170044Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host&X-Amz-Signature=958f36d2a1244a75072bb692cdbc5e5aaff7fead0f3742a2bbfb7d409ac058d1
Resolving github.s3.amazonaws.com (github.s3.amazonaws.com)... 52.216.228.96
Connecting to github.s3.amazonaws.com (github.s3.amazonaws.com)|52.216.228.96|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 180771 (177K) [application/x-gzip]
Saving to: ‘xar-1.6.1.tar.gz’


xar-1.6.1.tar.gz    100%[=====================>] 176.53K  --.-KB/s   in 0.003s 

2017-12-15 17:00:45 (53.8 MB/s) - ‘xar-1.6.1.tar.gz’ saved [180771/180771]

autoconf
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... no
checking how to run the C preprocessor... gcc -E
checking for a BSD-compatible install... /usr/bin/install -c
checking for ld... /usr/bin/ld
checking for ar... /usr/bin/ar
checking for ranlib... /usr/bin/ranlib
checking for autoconf... /usr/bin/autoconf
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for stdint.h... (cached) yes
checking ext2fs/ext2_fs.h usability... no
checking ext2fs/ext2_fs.h presence... no
checking for ext2fs/ext2_fs.h... no
checking sys/statfs.h usability... yes
checking sys/statfs.h presence... yes
checking for sys/statfs.h... yes
checking sys/xattr.h usability... yes
checking sys/xattr.h presence... yes
checking for sys/xattr.h... yes
checking sys/param.h usability... yes
checking sys/param.h presence... yes
checking for sys/param.h... yes
checking sys/extattr.h usability... no
checking sys/extattr.h presence... no
checking for sys/extattr.h... no
checking libutil.h usability... no
checking libutil.h presence... no
checking for libutil.h... no
checking for lgetxattr... yes
checking for lsetxattr... yes
checking for getxattr... yes
checking for setxattr... yes
checking for getattrlist... no
checking for setattrlist... no
checking for lchmod... no
checking for lchown... yes
checking for chflags... no
checking for statvfs... yes
checking for statfs... yes
checking for strmode... no
checking for struct statfs.f_fstypename... no
checking for struct statvfs.f_fstypename... no
checking for struct stat.st_flags... no
checking size of uid_t... 4
checking size of gid_t... 4
checking size of ino_t... 8
checking size of dev_t... 8
checking for acl_get_file in -lacl... no
checking for asprintf... yes
checking for xml2-config... /usr/bin/xml2-config
checking for libxml >= 2.6.11... 2.9.1
checking libxml/xmlwriter.h usability... yes
checking libxml/xmlwriter.h presence... yes
checking for libxml/xmlwriter.h... yes
checking openssl/evp.h usability... yes
checking openssl/evp.h presence... yes
checking for openssl/evp.h... yes
checking for OpenSSL_add_all_ciphers in -lcrypto... yes
checking zlib.h usability... yes
checking zlib.h presence... yes
checking for zlib.h... yes
checking for deflate in -lz... yes
checking bzlib.h usability... yes
checking bzlib.h presence... yes
checking for bzlib.h... yes
checking for BZ2_bzCompress in -lbz2... yes
checking lzma.h usability... yes
checking lzma.h presence... yes
checking for lzma.h... yes
checking for lzma_code in -llzma... yes
configure: creating ./config.status
config.status: creating cfgoutputs.stamp
config.status: creating Makefile
config.status: creating include/xar.h
config.status: creating lib/Makefile.inc
config.status: creating lib/libxar.la.in
config.status: creating src/Makefile.inc
config.status: creating xar.spec
config.status: creating cfghdrs.stamp
config.status: creating include/config.h
gcc -Wall -g -O2  -fPIC -DPIC -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/archive.c -o lib/archive.o
lib/archive.c: In function ‘xar_close’:
lib/archive.c:469:6: warning: variable ‘ret’ set but not used [-Wunused-but-set-variable]
  int ret, retval = 0;
      ^
lib/archive.c: In function ‘xar_unserialize’:
lib/archive.c:1684:12: warning: variable ‘noattr’ set but not used [-Wunused-but-set-variable]
  int type, noattr, ret;
            ^
lib/archive.c:1683:33: warning: variable ‘uri’ set but not used [-Wunused-but-set-variable]
  const xmlChar *name, *prefix, *uri;
                                 ^
lib/archive.c:1683:24: warning: variable ‘prefix’ set but not used [-Wunused-but-set-variable]
  const xmlChar *name, *prefix, *uri;
                        ^
gcc -Wall -g -O2  -fPIC -DPIC -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/arcmod.c -o lib/arcmod.o
gcc -Wall -g -O2  -fPIC -DPIC -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/b64.c -o lib/b64.o
gcc -Wall -g -O2  -fPIC -DPIC -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/bzxar.c -o lib/bzxar.o
gcc -Wall -g -O2  -fPIC -DPIC -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/darwinattr.c -o lib/darwinattr.o
gcc -Wall -g -O2  -fPIC -DPIC -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/data.c -o lib/data.o
gcc -Wall -g -O2  -fPIC -DPIC -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/ea.c -o lib/ea.o
gcc -Wall -g -O2  -fPIC -DPIC -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/err.c -o lib/err.o
gcc -Wall -g -O2  -fPIC -DPIC -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/ext2.c -o lib/ext2.o
gcc -Wall -g -O2  -fPIC -DPIC -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/fbsdattr.c -o lib/fbsdattr.o
gcc -Wall -g -O2  -fPIC -DPIC -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/filetree.c -o lib/filetree.o
lib/filetree.c: In function ‘xar_prop_next’:
lib/filetree.c:353:7: warning: variable ‘err’ set but not used [-Wunused-but-set-variable]
   int err;
       ^
lib/filetree.c:395:7: warning: variable ‘err’ set but not used [-Wunused-but-set-variable]
   int err;
       ^
lib/filetree.c: In function ‘xar_file_next’:
lib/filetree.c:850:8: warning: variable ‘err’ set but not used [-Wunused-but-set-variable]
    int err;
        ^
lib/filetree.c: In function ‘xar_prop_unserialize’:
lib/filetree.c:1058:10: warning: variable ‘err’ set but not used [-Wunused-but-set-variable]
      int err;
          ^
gcc -Wall -g -O2  -fPIC -DPIC -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/io.c -o lib/io.o
gcc -Wall -g -O2  -fPIC -DPIC -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/lzmaxar.c -o lib/lzmaxar.o
gcc -Wall -g -O2  -fPIC -DPIC -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/linuxattr.c -o lib/linuxattr.o
gcc -Wall -g -O2  -fPIC -DPIC -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/hash.c -o lib/hash.o
gcc -Wall -g -O2  -fPIC -DPIC -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/signature.c -o lib/signature.o
gcc -Wall -g -O2  -fPIC -DPIC -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/stat.c -o lib/stat.o
gcc -Wall -g -O2  -fPIC -DPIC -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/subdoc.c -o lib/subdoc.o
gcc -Wall -g -O2  -fPIC -DPIC -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/util.c -o lib/util.o
lib/util.c: In function ‘xar_get_path’:
lib/util.c:97:7: warning: variable ‘err’ set but not used [-Wunused-but-set-variable]
   int err;
       ^
gcc -Wall -g -O2  -fPIC -DPIC -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/zxar.c -o lib/zxar.o
gcc -Wall -g -O2  -fPIC -DPIC -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/script.c -o lib/script.o
gcc -Wall -g -O2  -fPIC -DPIC -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/macho.c -o lib/macho.o
gcc -shared -Wl,-soname,libxar.so.1 -o lib/librxar.so.1 lib/archive.o lib/arcmod.o lib/b64.o lib/bzxar.o lib/darwinattr.o lib/data.o lib/ea.o lib/err.o lib/ext2.o lib/fbsdattr.o lib/filetree.o lib/io.o lib/lzmaxar.o lib/linuxattr.o lib/hash.o lib/signature.o lib/stat.o lib/subdoc.o lib/util.o lib/zxar.o lib/script.o lib/macho.o -Llib  -llzma -lbz2 -lz -lcrypto  -lxml2
rm -f lib/librxar.so
ln -s librxar.so.1 lib/librxar.so
gcc -shared -Wl,-soname,libxar.so.1 -o lib/libxar.so.1 lib/archive.o lib/arcmod.o lib/b64.o lib/bzxar.o lib/darwinattr.o lib/data.o lib/ea.o lib/err.o lib/ext2.o lib/fbsdattr.o lib/filetree.o lib/io.o lib/lzmaxar.o lib/linuxattr.o lib/hash.o lib/signature.o lib/stat.o lib/subdoc.o lib/util.o lib/zxar.o lib/script.o lib/macho.o -Llib  -llzma -lbz2 -lz -lcrypto  -lxml2
rm -f lib/libxar.so
ln -s libxar.so.1 lib/libxar.so
gcc -Wall -g -O2   -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/archive.c -o lib/archive.static.o
lib/archive.c: In function ‘xar_close’:
lib/archive.c:469:6: warning: variable ‘ret’ set but not used [-Wunused-but-set-variable]
  int ret, retval = 0;
      ^
lib/archive.c: In function ‘xar_unserialize’:
lib/archive.c:1684:12: warning: variable ‘noattr’ set but not used [-Wunused-but-set-variable]
  int type, noattr, ret;
            ^
lib/archive.c:1683:33: warning: variable ‘uri’ set but not used [-Wunused-but-set-variable]
  const xmlChar *name, *prefix, *uri;
                                 ^
lib/archive.c:1683:24: warning: variable ‘prefix’ set but not used [-Wunused-but-set-variable]
  const xmlChar *name, *prefix, *uri;
                        ^
gcc -Wall -g -O2   -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/arcmod.c -o lib/arcmod.static.o
gcc -Wall -g -O2   -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/b64.c -o lib/b64.static.o
gcc -Wall -g -O2   -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/bzxar.c -o lib/bzxar.static.o
gcc -Wall -g -O2   -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/darwinattr.c -o lib/darwinattr.static.o
gcc -Wall -g -O2   -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/data.c -o lib/data.static.o
gcc -Wall -g -O2   -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/ea.c -o lib/ea.static.o
gcc -Wall -g -O2   -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/err.c -o lib/err.static.o
gcc -Wall -g -O2   -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/ext2.c -o lib/ext2.static.o
gcc -Wall -g -O2   -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/fbsdattr.c -o lib/fbsdattr.static.o
gcc -Wall -g -O2   -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/filetree.c -o lib/filetree.static.o
lib/filetree.c: In function ‘xar_prop_next’:
lib/filetree.c:353:7: warning: variable ‘err’ set but not used [-Wunused-but-set-variable]
   int err;
       ^
lib/filetree.c:395:7: warning: variable ‘err’ set but not used [-Wunused-but-set-variable]
   int err;
       ^
lib/filetree.c: In function ‘xar_file_next’:
lib/filetree.c:850:8: warning: variable ‘err’ set but not used [-Wunused-but-set-variable]
    int err;
        ^
lib/filetree.c: In function ‘xar_prop_unserialize’:
lib/filetree.c:1058:10: warning: variable ‘err’ set but not used [-Wunused-but-set-variable]
      int err;
          ^
gcc -Wall -g -O2   -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/io.c -o lib/io.static.o
gcc -Wall -g -O2   -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/lzmaxar.c -o lib/lzmaxar.static.o
gcc -Wall -g -O2   -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/linuxattr.c -o lib/linuxattr.static.o
gcc -Wall -g -O2   -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/hash.c -o lib/hash.static.o
gcc -Wall -g -O2   -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/signature.c -o lib/signature.static.o
gcc -Wall -g -O2   -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/stat.c -o lib/stat.static.o
gcc -Wall -g -O2   -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/subdoc.c -o lib/subdoc.static.o
gcc -Wall -g -O2   -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/util.c -o lib/util.static.o
lib/util.c: In function ‘xar_get_path’:
lib/util.c:97:7: warning: variable ‘err’ set but not used [-Wunused-but-set-variable]
   int err;
       ^
gcc -Wall -g -O2   -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/zxar.c -o lib/zxar.static.o
gcc -Wall -g -O2   -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/script.c -o lib/script.static.o
gcc -Wall -g -O2   -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c lib/macho.c -o lib/macho.static.o
/usr/bin/ar cvr lib/libxar.a lib/archive.static.o lib/arcmod.static.o lib/b64.static.o lib/bzxar.static.o lib/darwinattr.static.o lib/data.static.o lib/ea.static.o lib/err.static.o lib/ext2.static.o lib/fbsdattr.static.o lib/filetree.static.o lib/io.static.o lib/lzmaxar.static.o lib/linuxattr.static.o lib/hash.static.o lib/signature.static.o lib/stat.static.o lib/subdoc.static.o lib/util.static.o lib/zxar.static.o lib/script.static.o lib/macho.static.o
a - lib/archive.static.o
a - lib/arcmod.static.o
a - lib/b64.static.o
a - lib/bzxar.static.o
a - lib/darwinattr.static.o
a - lib/data.static.o
a - lib/ea.static.o
a - lib/err.static.o
a - lib/ext2.static.o
a - lib/fbsdattr.static.o
a - lib/filetree.static.o
a - lib/io.static.o
a - lib/lzmaxar.static.o
a - lib/linuxattr.static.o
a - lib/hash.static.o
a - lib/signature.static.o
a - lib/stat.static.o
a - lib/subdoc.static.o
a - lib/util.static.o
a - lib/zxar.static.o
a - lib/script.static.o
a - lib/macho.static.o
/usr/bin/ranlib lib/libxar.a 
sed -e s/@LIBXAR_SNAME@/libxar.so.1/ -e s/@LIBXAR_LNAME@/libxar.so/ -e s/@LIBXAR_ANAME@/libxar.a/ < lib/libxar.la.in > lib/libxar.la
gcc -Wall -g -O2  -Iinclude -Iinclude -D_GNU_SOURCE -I/usr/include/libxml2 -c src/xar.c -o src/xar.o
src/xar.c: In function ‘is_valid_dir’:
src/xar.c:672:6: warning: variable ‘err’ set but not used [-Wunused-but-set-variable]
  int err, result, curdir = open(".", O_RDONLY);
      ^
src/xar.c: In function ‘archive’:
src/xar.c:1211:7: warning: variable ‘err’ set but not used [-Wunused-but-set-variable]
   int err;
       ^
src/xar.c: In function ‘main’:
src/xar.c:1931:6: warning: variable ‘ret’ set but not used [-Wunused-but-set-variable]
  int ret;
      ^
gcc -Wall -g -O2  -o src/xar src/xar.o -Wl,-rpath,/tmp/download/xar-1.6.1/lib -Llib  lib/libxar.a -llzma -lbz2 -lz -lcrypto  -lxml2
gcc -Wall -g -O2  -o src/ixar src/xar.o -Wl,-rpath,/usr/local/lib -Llib  lib/libxar.so.1 -llzma -lbz2 -lz -lcrypto  -lxml2
/usr/bin/install -c -d /usr/local/include/xar
/usr/bin/install -c -m 0644  include/xar.h /usr/local/include/xar
/usr/bin/install -c -d /usr/local/lib
/usr/bin/install -c -m 0755 lib/libxar.so.1 /usr/local/lib
rm -f /usr/local/lib/libxar.so
ln -s libxar.so.1 /usr/local/lib/libxar.so
/usr/bin/install -c -m 0644 lib/libxar.a /usr/local/lib
/usr/bin/install -c -m 0644 lib/libxar.la /usr/local/lib
/usr/bin/install -c -d /usr/local/bin
/usr/bin/install -c -m 0755 src/ixar /usr/local/bin/xar
/usr/bin/install -c -d /usr/local/share/man/man1
/usr/bin/install -c -m 0644 src/xar.1 /usr/local/share/man/man1
``

But this “feels” a bit wrong.

Is this the best way to do this?
Also, can I somehow cache the result?


#2

You could either make a package yourself and put in in a PPA so that you don’t have to compile each time.

Or you can extend the docker image to do this step once and then use that same docker image with this already installed.


#3

Building a package and putting it in a PPA turned out a bit more involved than I hoped. Very Linux-y :wink:

Creating a new Docker image seems the way to go, I will look into this.

Thanks!


#4

It is a little more involved but you get the added benefit of doing a public service so that other developers don’t have to jump through this same hoop in the future. :slight_smile:

But yeah, generally maintaining a Debian package is not something that I would consider fun.


#5

This topic was automatically closed 41 days after the last reply. New replies are no longer allowed.