I enabled ‘Only build pull requests’, which is what I hoped for, the issue now is that everything I push a changes, and while I did not create a pull request for it, CircleCI will still detect it and create a ‘Not Run’ job.
There is no way to do either of those things. This seems like a great idea though, I would encourage you to open up a feature request on our ideas board: https://circleci.com/ideas/