Python Set Version takes 2-3 minutes per build

Seems like from previous discourse on this topic CircleCI downloads python every time to the container and installs it. Is there any way to cache this process? People spoke of using pyenv, which I played with for a bit but couldn’t get it to work. I also wasn’t sure how to make sure the pyenv version of python was executing the commands