Is there a way to build all PRs to specific branches and those branches themselves?

Could you run builds on master and also a special tag trigger? It is possible to limit builds to a set of branches and named tags.

Your workflow in the PR branch would be to add the tag to the head, delete that tag remotely if it exists, and then push the tag with git push --follow-tags. That would be a few lines of Bash scripting, run locally.