Do (or could) all of your five jobs run on the same base image? If so, and if the jobs are in the same workflow, you could use workspaces to share folders of data between them, so that only the first job has to build the environment.
Alternatively, if the images need lots of things installing, then you could create a separate pipeline to build your base image, and just rebuild it weekly. For this approach, push it to an external (public/private) registry and then pull it in CircleCI. It is also a good idea to use a lightweight OS (such as Alpine) in order to reduce the size of images (100M is much better than 1.5G!).
Finally if you are building images and want a variety of layer caching options, consider this post.