Multiple deploy keys for GitHub

[JanBrinker]

Hi there,
I know, there are multiple threads about this kind of issue already, but all of them are closed and there’s no option of asking follow up questions.

So, the situation is like this: to build our application we need to access a second repository located on the same host as the project repo itself (GitHub). Now with GitHub we can’t use the same deploy keys for multiple repos. On Bitbucket this works, but alas, we’re on GitHub. Machine users are out of question as some organisation rules don’t allow us to create machine users. For the same reasons we also should not use user keys.
I was looking at something like this: https://gist.github.com/jamesmcfadden/d379e04e7ae2861414886af189ec59e5, (using host aliases via ~/.ssh/config) but it seems like this won’t work with CircleCI. Any ideas on how to solve this?

[halfer]

The manual setting up of repo cloning is solved for GitHub here:

Cloning another private repo in the build

Hi guys, I have a question regarding cloning another private repository. Let’s say I have two repositories: my-app-backend and my-app-frontend. When going through the workflow with my-app-backend, during one step, I want to clone the frontend repository to run some tests. I’ve set up SSH keys already, namely: I generated new public and private SSH keys. The repository is on Github so I visited the frontend repo, Settings -> Deploy keys and according to hints I added the public SSH key which wa…

However, I wonder if I am misunderstanding you - what are deploy keys in the context of GitHub? One cannot deploy to GitHub.

[JanBrinker]

The post you mentioned refers to the documentation which is this: https://circleci.com/docs/2.0/gh-bb-integration/#enable-your-project-to-check-out-additional-private-repositories and there it recommends user keys. As I stated above, in my organization we were asked not to use user keys, so I’m looking for alternatives.

About your question: GitHub deploy keys could also be named “Checkout SSH keys”, but the principle is the same with deploy keys: allowing access (read or write, depending in permissions) to a resource via SSH

[halfer]

Ooh, “Deploy Keys” in a GitHub per-repo thing, who knew! Right, sorry

OK, so yes, this will work in CircleCI for cloning a secondary repo. If the core problem is you can’t use duplicate SSH keys in GH, then create a new “Deploy” key in GitHub, and add the other side of the key in Checkout SSH Keys in your per-project CircleCI config.

it seems like this won’t work with CircleCI

Why not? This ought to work fine, subject to the usual fiddle that PPK often creates!

[JanBrinker]

After adding the checkout (deploy) key for the project in CircleCI I don’t see any option of adding another one. Furthermore from what I’ve gathered on previous posts on this board CircleCI will have two keys, both pointing to github.com. CircleCI won’t know which SSH key to use for which repo and will try to use either the first one for both repos or the second one for both repos. Obviously, in both cases one of the repos can’t be accessed… Theoretically could be mitigated by modifying the .ssh/config file and using host aliases (see my link in the original post), but from what I’ve read this is not possible. Right?

[halfer]

There’s a lot of missing information here, so I’m in danger of making too many guesses. Would you supply your config.yml file?

I assume your primary checkout just uses the checkout command, and your secondary checkout uses a git clone command. You will also need the YAML command add_ssh_keys to copy all your checkout keys into place.

Yes, use Host aliases. The main gist you’ve linked to overly confuses the issue, and the replies are much better. Here is what I use for a server alias (on my local machine, but the principle is the same):

Host agnes
    Hostname 45.77.54.171
    IdentityFile ~/.ssh/vultr_rsa
    IdentitiesOnly yes

As you can see, the Host is any old nonsense, and it is resolved by the Hostname. When I SSH to agnes I use root@agnes, I imagine you can do much the same for a git clone. If GitHub does not recognise this, you can specify which key to use using the SSH switch to the git command.

[halfer]

After adding the checkout (deploy) key for the project in CircleCI I don’t see any option of adding another one.

Hmm, good point (I did not know that either!). Can you use “SSH Permissions” instead? If not, you should be able to use “Environment Variables” - it is, after all, just a string. You would need to do your own copy into place, but that’s trivial.

If I were tackling this issue, I’d get an SSH session and tinker away until I got a clone working on a secondary key. Just remember that SSH eats your build minutes, so be brisk about it, and remember to kill the session in the UI after you have disconnected.

[JanBrinker]

I had some time to test a few things. I wasn’t able to get things up and running via host aliases yet, but I found another embarassingly simple solution that is less versatile but works fine at least if you have only one more repo that you need to access:

1. Generate the SSH deploy key for GitHub and add it there. (Make sure it’s without password)

2. Add it to CircleCI settings via the web interface under the section “SSH Keys” and set the host to “github.com” (or wherever your repo lies)
   When defining your job in the config.yml do this, in the exact order:

    steps:
      - checkout # First check out your repo with the given SSH checkout key
      - add_ssh_keys: # THEN set the *new* SSH key for github.
          fingerprints:
              - "<your fingerprint>"
   

So basically after the first checkout you tell CircleCI to use a different SSH key for GitHub for the next requests. This works well for accessing one more repo. However things will get really messy when you try to use the same method for accessing a third, fourth… repo. For that you will still need to use aliases. When I have the time I will try to find a solution for that aswell. But in the meanwhile, this works nice.

This might be especially interesting for other iOS projects like us that are using fastlane match.

[Vadimkomis]

Can you give an example of your config.yml file?
I’m trying to do the same exact thing, and it keeps telling me that
“^D^Dfatal: destination path ‘{project_name}’ already exists and is not an empty directory.”

My objective is to run local host (one repository) of Ruby on Circle and and then run smoke tests (iOS build from another repository) of the local host.

[JanBrinker]

I guess something else is wrong with your config.yml. Sadly I cannot share our config, though…

[brockfanning]

@JanBrinker, I’m excited to try that approach, which sounds like it would work for our use-case: using the main repository for “staging”, while having a job that deploys to a separate “production” repository in a different Github organisation.

So far we have been using a pretty hacky approach to accomplish this, by manually changing the .ssh/config file. In a nutshell, the main steps are:

1. In the “SSH Permissions” tab on CircleCI.com, add 2 separate SSH keys, one with a hostname of “gh-stg” and the other with a hostname of “gh-prod”.
2. In the workflow when the staging repository needs to be cloned, use this code:
   • sed -i -e 's/Host gh-stg/Host gh-stg\n HostName github.com/g' ~/.ssh/config
   • git clone git@gh-stg:my-staging-github-org/my-staging-repo.git
3. Likewise, when the production repository needs to be cloned, use this code:
   • sed -i -e 's/Host gh-prod/Host gh-prod\n HostName github.com/g' ~/.ssh/config
   • git clone git@gh-prod:my-prod-github-org/my-prod-repo.git

[brockfanning]

After thinking more on it I don’t believe JanBrinker’s approach would work in my particular use-case. If I’m understanding correctly, that will allow you to: read from repo #1, and then read/write to repo #2. But in my case I’m looking to read/write to both repo #1 and repo #2. So I believe I’m stuck with the hacky approach I posted above. Please let me know if I’m misunderstanding.