Self-hosted cache?

Build Environment Caching Dependencies
1

Hi,
I have a client with an office location in Australia. When setting up their self-hosted runner I noticed that caching(saving and restoring) takes incredibly long time.
After running traceroute to caching storage it start to make sense:

traceroute  circle-production-customer-artifacts.s3.amazonaws.com
traceroute: Warning: circle-production-customer-artifacts.s3.amazonaws.com has multiple addresses; using 52.216.61.169
traceroute to s3-w.us-east-1.amazonaws.com (52.216.61.169), 64 hops max, 52 byte packets
 1  192.168.0.1 (192.168.0.1)  3.093 ms  3.986 ms  2.853 ms
 2  bdr01-pex-20wharfs-bne.au.superloop.net.co (27.122.112.255)  7.419 ms  6.809 ms  6.917 ms
 3  gigabitethernet0-0-0-7.bdr01-ipt-20wharfs-bne.au.superloop.net.co (103.200.14.240)  7.318 ms  6.658 ms  6.740 ms
 4  bundle-ether33.bdr02-ipt-20wharfs-bne.au.superloop.net.co (103.200.13.98)  166.114 ms  167.389 ms  166.207 ms
 5  bundle-ether34-102.bdr02-ipt-47bourke-syd.au.superloop.net.co (103.200.13.109)  166.566 ms
    bundle-ether34-103.bdr02-ipt-4edenpar-syd.au.superloop.net.co (103.200.13.224)  167.401 ms
    bundle-ether34-102.bdr02-ipt-47bourke-syd.au.superloop.net.co (103.200.13.109)  166.932 ms
 6  bundle-ether30.bdr02-ipt-639garde-syd.au.superloop.net.co (103.200.13.65)  173.178 ms
    bundle-ether32.bdr02-ipt-639garde-syd.au.superloop.net.co (103.200.13.70)  166.204 ms
    bundle-ether30.bdr02-ipt-639garde-syd.au.superloop.net.co (103.200.13.65)  168.281 ms
 7  hundredgige0-0-1-2-116.bdr01-ipt-55smarke-sjc.us.superloop.net.co (103.200.13.249)  167.222 ms  168.157 ms  167.261 ms
 8  sjo-b23-link.ip.twelve99.net (213.248.68.126)  167.474 ms  166.130 ms  166.766 ms
 9  palo-b24-link.ip.twelve99.net (62.115.115.216)  168.272 ms  168.358 ms  168.587 ms
10  lax-b22-link.ip.twelve99.net (62.115.119.91)  174.108 ms  172.117 ms  174.375 ms
11  * * dls-bb1-link.ip.twelve99.net (62.115.118.246)  203.187 ms
12  vadata-svc071233-lag003357.ip.twelve99-cust.net (62.115.61.163)  203.190 ms  236.598 ms  212.267 ms
13  vadata-svc071233-lag003357.ip.twelve99-cust.net (62.115.61.163)  215.341 ms  203.162 ms  221.034 ms
14  150.222.206.207 (150.222.206.207)  212.373 ms  212.953 ms
    150.222.206.209 (150.222.206.209)  219.714 ms
15  15.230.48.36 (15.230.48.36)  216.601 ms
    176.32.125.195 (176.32.125.195)  212.279 ms
    15.230.48.48 (15.230.48.48)  213.291 ms
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  52.93.29.80 (52.93.29.80)  231.494 ms
    52.93.29.68 (52.93.29.68)  231.830 ms
    52.93.29.92 (52.93.29.92)  233.310 ms
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  s3-1-w.amazonaws.com (52.216.61.169)  234.451 ms  234.104 ms  233.441 ms

We have large deps like ios build cache(700MB) and under such network conditions the build will probably take hours.
I wonder if there is a way to self-hosted cache or maybe there is another strategy that can be used I cannot think of?

Thank you!

2

When working with a self-hosted runner you can script just about anything you want as unlike a CircleCI-provided environment your runner’s environment is not wiped at the end of a build job.

The starting points are

  • make use of the ‘working_directory’ parameter for the defined executor. This allows you to define a static/unique directory for each job you wish to run. With this setup, you now have a local area that you can use to cache files.

  • consider a dedicated runner for each possible job. This is just a simple way to make sure you do not have to move copies of a unique working directory between runners.

- make sure you give your runner a lot of free disk space.

Within my own environment doing the above gives me

  • docker cache as it is just a feature of docker. All I need to do is run a docker prune to limit the number of images being held.
  • The CircleCI checkout step is coded to do a git clone/fetch based on what it finds in the working directory.
  • anything else I just leave in the directory.

There is a lot more I could script up to improve performance and adding an HTTP proxy would improve things, but it is currently good enough for me.

1 Like
3

Thanks for the reply!
Curios are you using container or machine self-hosted runner?

4

Currently, I’m using machine based self-hosted runners. The reason for this just comes down to the fact that all the developer written scripts I moved across to the CI system at the start of the project just fitted the machine model and I’ve not yet found a reason to change anything.