Cannot SSH to my server from a job, but it's ok in SSH debug

I’m trying to use cimg/base:stable-20.04 for my deploy jobs. I cannot ssh to my servers from my jobs but I can do it on SSH debug session.

During the job execution:

ssh -v vincent@$ip "echo a"
OpenSSH_8.2p1 Ubuntu-4ubuntu0.1, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /home/circleci/.ssh/config
debug1: /home/circleci/.ssh/config line 2: Applying options for *
debug1: /home/circleci/.ssh/config line 6: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to X.X.X.X [X.X.X.X] port 22.
debug1: connect to address X.X.X.X port 22: Connection refused
ssh: connect to host X.X.X.X port 22: Connection refused
Exited with code exit status 255
CircleCI received exit code 255

On an SSH debug session:

circleci@324ad8ad87f4:~$ ssh vincent@X.X.X.X
Welcome to Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-1037-aws x86_64)

If I check /var/log/auth.log on my server, that is running on ubuntu 20.04, I get the following error when the CircleCI job is trying to connect:

    Feb  9 09:30:01 ip-XXX sshd[190606]: Connection closed by 3.91.195.239 port 54434 [preauth]
    Feb  9 09:30:01 ip-XXX sshd[190609]: Connection closed by 3.91.195.239 port 54436 [preauth]
    Feb  9 09:30:02 ip-XXX sshd[190607]: Connection closed by 3.91.195.239 port 54438 [preauth]
    Feb  9 09:30:02 ip-XXX sshd[190610]: Unable to negotiate with 3.91.195.239 port 54440: no matching host key type found. Their offer: sk-ecdsa-sha2-nistp256/@openssh.com [preauth]
    Feb  9 09:30:02 ip-XXX sshd[190608]: Unable to negotiate with 3.91.195.239 port 54442: no matching host key type found. Their offer: sk-ssh-ed25519/@openssh.com [preauth]

I’m completely certain of the SSH key used since the same job was working with a homemade docker image base on ubuntu 18.04