Circleci self hosted runner cache is very slow

Build Environment
1

circleci self hosted runner cache is very slow

cache / persistence workspace on circleci takes 20 seconds

but it takes 12 minutes on self hosted runners on k8s

2

It can be, as you are transferring data to and from a system that is located within the data centers that CircleCI uses for its deployed infrastructure. These will not be optimized for your runners.

If you are running self-hosted runners it is normally better to deploy your own persistence solution. I use VMs for my self-hosted runners and they just use local storage as a way to persist data between jobs and runs.

3

is there some blog post of how to do it?

how can I cache directly in my kubernetes?

4

I do not know much about k8s, but for my VM based system all I do is define the executor like this

executors:
  exec_node:
    machine: true
    working_directory: /tmp/CircleCI_working_directories/Build-Frontend
    resource_class: company/basic-linux

So this is the executor deployed as part of the frontend workflow. This has only a single defined job, which will always be run on the named hosted runner “company/basic-linux” and within the defined working directory.

As I’ve allocated the VM 500GB of storage space there is lots of free space for data to be placed in the working directory and retained between runs.

So if you can define a k8s runner with persistent storage you can do the same thing.

5

not sure if I can have persistent storage on the k8s

6

An alternative solution is to use nonpersistent runners and some form of other storage such as S3 (in-house or at least nearer than CircleCI’s offering)

The dedicated CircleCI commands don’t do much more than store/retrieve compressed directory trees so you can do much the same with installed copies of 7z and rclone.

You could even use a git repo as a storage location depending on the git solution you use.

7

can I check the source of save cache and restore cache? and persistence workspace somewhere else?

maybe a orb to save this complexity ?

8

The commands are core parts of the circle.yml processor, so I am not aware of the source being available and with them being added to the system the development of ORBs to do the same thing by others stopped.