Publishing Orb with non-owner github user

I have created my orb and namespace using my github user which is org admin on github. I have a new github user (with a member role) that is gonna be used for automatic orb publishing using GitHub Action, however I get this error when I want to publish the orb with this new user:

User does not have access to publish SemVer orbs in this namespace.

I can see in the circleci orb documentation and also some topics here that only owner can publish an orb with prod version.
I don’t want to give this user admin access to the github org. As this user is only would be used for this purpose and doesn’t need to have access to everything on the org. Is there anyway to give this user access to publish the orb?
What is the best way for this case?
How is it possible to change the ownership of an orb and namespace?

Any help would be appreciated.