Multiple workspaces / drop workspace layers between jobs?



I’m building a Node app, packaging it in a Docker container and pushing/deploying to Google Cloud.
I’m trying to break those steps cleanly into jobs (so that I can use standard base docker images and not have to build a custom one with everything), but persisting everything to workspaces and attaching is slow.

What would be the best way to persist specific directories / artifacts between the jobs below ?

The first 3 jobs are based on “circleci/node”:

  • checkout and modules : checkout code + build node_modules dependencies (with caching), persist everything to workspace
  • testing : needs the whole workspace
  • pack / build once tests have passed : perform final babel transpilation + copy various files all into a “dist” subdirectory. Needs the whole workspace one last time but next steps only require the content of “dist”.
    From then on the base image is “google/cloud-sdk”
  • docker build + push to Google Container Registry (hence the need for gcloud sdk) : only need the “dist” directory from the previous workspace, not the whole source tree ! I need to do the build+push in the same job because the remote docker environment isn’t persisted between steps.
  • deploy to kubernetes : doesn’t require anything else because the container is in the registry already.

So, I don’t need to persist the whole workspace with source tree beyond the “pack / build” job but I do need to carry across a few directories/files between jobs :

  • Would it be a good idea to abuse the save_cache feature to persist only the directories/files I need (with a unique key based on CIRCLE_SHA1) for the next job. I say “abuse” because it’s not really a cache: I can’t afford NOT to have the data from the previous job, and I also don’t need it beyond this build.

  • I could use “docker save” to turn the docker image into an artifact at the end of the docker build job instead of pushing it (so that I don’t need the Google SDK for registry access), and then “docker load” it in the next job which would have the Google SDK and would push and deploy. Are artifacts designed for that though, can you read artifacts from a previous job in the next one ?



This topic was automatically closed 41 days after the last reply. New replies are no longer allowed.