Google Container Engine Deployment Fails

Feedback & Bug Reports
,
1

I have a Google Container Engine (Kubernetes) app set up to build and deploy on CircleCI 1.0. Deployment was working before by patching the deployment but now the kubectl command fails with

Unable to connect to the server: error executing access token command “/opt/google-cloud-sdk/bin/gcloud config config-helper --format=json”: err=exit status 1 output=

Nothing has changed in the environment since the last successful deployment; the kubectl client is at v1.6.0 and the server is using v1.5.6. Running the same deployment command locally using the same authentication private key works without a hitch, and I can successfully ping the server IP from the CircleCI build container so I’m not sure what to do next to debug this error message…

Additionally, after running a kubectl command I run into a new error, where I get logged out of the container and attempting to reconnect warns me that the remote host identification has changed somehow, even if the build is still running and the machine should be available.

I…I don’t even know anymore.

4 Likes
2

I’m getting the same issue. Tried running kubectl cluster-info and it fails in accessing the server. No change was made in regards to credentials etc.

3

Same for me, it worked a day and a half ago and now it doesn’t work on all my projects

1 Like
4

Same for us with https://github.com/akvo/akvo-lumen

1 Like
5

This just happened to me, following an update on my local machine of the glcoud SDK, from version 146.0.0 to version to 150.0.0. I reverted and all is back to normal. To revert I used:

gcloud components update --version 146.0.0

Hope that helps!

1 Like
6

Worked like a charm. Thanks.

It seems that the current version installed by default in the circleci container is 149.0.0 and that also has some issues (not just 150.0.0).

Since 150.0.0 kubectl works well from my local machine onto the same cluster I assume it’s a combination of the kubectl version and something on the circleci side.

2 Likes
7

Any update here on this issue?

8

I ended up just rewriting everything to CircleCI 2.0 format and using a custom Docker container that extended Node 7.3 to install gcloud tools. That worked pretty well actually.

9

tl;dr see the last command for the solution

The error suggests something is wrong with the "/opt/google-cloud-sdk/bin/gcloud config config-helper --format=json command. I indeed ran this command in my CircleCI test and got back error:

WARNING: The credentials file (/home/ubuntu/.config/gcloud/credentials) is not writable. Opening in read-only mode. Any refreshed credentials will only be valid for this run.
ERROR: (gcloud.config.config-helper) Your current active account [coffeelog-circleci-cd@xxx.iam.gserviceaccount.com] does not have any valid credentials
Please run:

  $ gcloud auth login

to obtain new credentials, or if you have already logged in with a
different account:

  $ gcloud config set account ACCOUNT

to select an already authenticated account to use.

It appears like if you are running gcloud auth activate-service-account as sudo (which is what the Circle CI tutorial suggests), the file /home/ubuntu/.config/gcloud/credentials file will have root:root owner (not ubuntu:ubuntu).

Therefore your gcloud command (which calls gcloud config config-helper) can’t read the credentials…

To fix this error, run the following, which will make the gcloud credentials file readable without sudo:

sudo chown -R $USER /home/ubuntu/.config

You might be wondering, can’t I just do sudo kubectl, it looks like root user doesn’t have kubectl in PATH. I think the command above is enough to fix it.

Also, if you are just building container images, give Google Container Builder. I have a blog post here: https://ahmetalpbalkan.com/blog/container-builder/

2 Likes
10

Okay that’s a really cool fix, kudos for finding this! Should probably update the CircleCI tutorial repo to reflect this discovery right?

As for GCB: it’s tempting, but does it support doing things like running tests? That’s my current major use case for CircleCI, a test + subsequent build

11

My experience with this is that the gcloud upgrade broke kubectl. After upgrading kubectl to 1.6.1 I think I’m back to sanity.

12

Thank you @ahmetalpbalkan for your solution!

We’re looking into ways to improve this experience, please bear with us :bow:

1 Like
13

We can confirm that @ahmetalpbalkan’s workaround solves the issue for us.

1 Like
14

I have the same problem ,but i had solved this problem by the next step:

$ rm $(which kubectl)
$ gcloud components install kubectl
15

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.