I have looked at this with a fresh pair of eyes and can confirm that this is not an issue, SSH access is restricted to the SSH keys that I have stored in GitHub, e.g. https://github.com/gerhard.keys:
You can now SSH into this box if your SSH public key is added:
$ ssh -p 64537 184.108.40.206
Use the same SSH public key that you use for your VCS-provider (e.g., GitHub).
RSA key fingerprint of the host is
SSH Session Activity Log (times in UTC):
Jan 17 09:29:07 Authorizing gerhard with MD5:0a:cc:dd:82:23:46:98:a4:2b:69:3c:9d:45:c2:c8:aa
Jan 17 09:29:07 Authorizing gerhard with MD5:e5:0a:b6:18:5f:45:6c:50:57:28:cc:26:94:96:c6:9d
Jan 17 09:29:07 Authorizing gerhard with MD5:ce:be:71:c9:32:27:32:cf:c6:d1:6c:11:46:42:b8:9a
I confirmed by trying to SSH from a host where I do not have access to my SSH keys:
Could not open a connection to your authentication agent.
ssh -p 64537 220.127.116.11
The authenticity of host '[18.104.22.168]:64537 ([22.214.171.124]:64537)' can't be established.
RSA key fingerprint is SHA256:vtBxs9klpcNvhnj8+wxjapGxzVhiMsXwcF6uRUN4XCU.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[126.96.36.199]:64537' (RSA) to the list of known hosts.
firstname.lastname@example.org: Permission denied (publickey).