Hi there, I’ve been using circleci 1.0 to run a simple pipeline, which did something like this:
spark-submit --driver-memory 126G /opt/my-spark-script.py"
ie it first runs a custom script to download some data and then runs two scripts over that data, each in their own docker container.
I feel this should be a lot cleaner in circleci2.0 but i’m confused how the docker executors work? They are started in the “spinup env” step, so should i create three jobs for the above? What is the recommended way to achieve the same thing, but rely on circleci’s new docker support (so it doesn’t have to fetch the docker images each run).
Or should i just run my pipeline.sh as is?