Following up here, this is now available self-serve via the API [Product Launch] Override Only Build PRs Setting