Gcloud permissions error

docker
circle.yml

#1

I’m trying to do some kubernetes commands as part of my deployment process, but I’m getting permission errors when trying to update gcloud tools.

Following this guide (https://circleci.com/docs/google-cloud-platform), I’ve added

dependencies: 
      override:
        - gcloud --quiet components update kubectl

This example project here also implies that it should work:

I am, however, just getting this error:

ERROR: (gcloud.components.update) You cannot perform this action because you do not have permission to modify the Google Cloud SDK installation directory [/opt/google-cloud-sdk].

Any ideas?


#2

Same here…
gcloud --quiet components update app
seems to work without problems…


#3

/opt is not owned by the ubuntu user so you need to run the command as sudo

However does @MichaelW’s suggestion work for you as well?


#4

I just tried

sudo gcloud --quiet components update

and got:

sudo: gcloud: command not found Action failed:


#5

Why is gcloud manually installed into a location that isn’t in root’s PATH? Why not install it from debian’s so that I can just sudo apt-get update instead?


#6

Well, that failed as well:

sudo /opt/google-cloud-sdk/bin/gcloud --quiet components update

./deploy.sh


Your current Cloud SDK version is: 98.0.0
You will be upgraded to version: 100.0.0

┌────────────────────────────────────────────────────────┐
│           These components will be updated.            │
├─────────────────────────────────┬────────────┬─────────┤
│               Name              │  Version   │   Size  │
├─────────────────────────────────┼────────────┼─────────┤
│ Cloud SDK Core Libraries        │ 2016.03.04 │ 3.9 MiB │
│ Cloud Storage Command Line Tool │       4.17 │ 2.6 MiB │
└─────────────────────────────────┴────────────┴─────────┘

For the latest full release notes, please visit:
  https://cloud.google.com/sdk/release_notes

╔════════════════════════════════════════════════════════════╗
╠═ Creating update staging area                             ═╣
╠════════════════════════════════════════════════════════════╣
╠═ Uninstalling: Cloud SDK Core Libraries                   ═╣
╠════════════════════════════════════════════════════════════╣
╠═ Uninstalling: Cloud Storage Command Line Tool            ═╣
╠════════════════════════════════════════════════════════════╣
╠═ Installing: Cloud SDK Core Libraries                     ═╣
╠════════════════════════════════════════════════════════════╣
╠═ Installing: Cloud Storage Command Line Tool              ═╣
╠════════════════════════════════════════════════════════════╣
╠═ Creating backup and activating new installation          ═╣
╚════════════════════════════════════════════════════════════╝


Performing post processing steps...done.

Update done!

To revert your SDK to the previously installed version, you may run:
  $ gcloud components update --version [98.0.0]

Traceback (most recent call last):
  File "/opt/google-cloud-sdk/lib/gcloud.py", line 53, in <module>
    main()
  File "/opt/google-cloud-sdk/lib/gcloud.py", line 49, in main
    sys.exit(googlecloudsdk.gcloud_main.main())
  File "/opt/google-cloud-sdk/lib/googlecloudsdk/gcloud_main.py", line 167, in main
    gcloud_cli = CreateCLI([])
  File "/opt/google-cloud-sdk/lib/googlecloudsdk/gcloud_main.py", line 121, in CreateCLI
    generated_cli = loader.Generate()
  File "/opt/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 344, in Generate
    cli = self.__MakeCLI(top_group)
  File "/opt/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 546, in __MakeCLI
    log.AddFileLogging(self.__logs_dir)
  File "/opt/google-cloud-sdk/lib/googlecloudsdk/core/log.py", line 656, in AddFileLogging
    _log_manager.AddLogsDir(logs_dir=logs_dir)
  File "/opt/google-cloud-sdk/lib/googlecloudsdk/core/log.py", line 364, in AddLogsDir
    file_handler = logging.FileHandler(log_file)
  File "/usr/lib/python2.7/logging/__init__.py", line 903, in __init__
    StreamHandler.__init__(self, self._open())
  File "/usr/lib/python2.7/logging/__init__.py", line 928, in _open
    stream = open(self.baseFilename, self.mode)

./deploy.sh returned exit code 1

IOError: [Errno 13] Permission denied: '/home/ubuntu/.config/gcloud/logs/2016.03.16/02.00.03.519350.log' Action failed: ./deploy.sh

I’m going to go back to my original working path of just installing the debian and pointing to that path.

# Install gcloud cause the circle server already has an old version of it installed: https://cloud.google.com/sdk/#deb
# Create an environment variable for the correct distribution
export CLOUD_SDK_REPO="cloud-sdk-$(lsb_release -c -s)"
# Add the Cloud SDK distribution URI as a package source
echo "deb http://packages.cloud.google.com/apt $CLOUD_SDK_REPO main" | sudo tee /etc/apt/sources.list.d/google-cloud-sdk.list
# Import the Google Cloud public key
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
# Update and install the Cloud SDK
sudo apt-get update && sudo apt-get install google-cloud-sdk

export PATH="/usr/lib/google-cloud-sdk/bin:$PATH"


Gcloud authentication fails
#7

My experience mirrors that of @lookfirst.

I’ve tried this:
sudo /opt/google-cloud-sdk/bin/gcloud --quiet components update kubectl
I.e. just sudo is not enough, you need to provide the absolute path to the gcloud binary.

Doing this completes the update, BUT when actually trying to do something with kubernetes (for example the necessary gcloud auth activate-service-account --key-file ${HOME}/client-secret.json) after that will break. With the same error as @lookfirst gets, i.e. not permitted to write to the logs.

This is pretty bad. Deploying to kubernetes on GCP has to be a pretty common use case down the line, and this breaks the happy path for that.

@levlaz I don’t think the post by MichaelW was a suggestion for solution. I think he meant that he had the same problem as I do, but that installing the app engine tools (gcloud --quiet components update app) worked for him, but installing the kubernetes tooling (gcloud --quiet components update kubectl) does not, and that he found that peculiar. For me, I don’t think that actually works, because when I run that on circleCI I get this message:
“WARNING: Component [app] no longer exists.”
… because google are now including app engine tooling as part of the gcloud tooling: http://stackoverflow.com/questions/33971523/google-cloud-component-app-no-longer-exist
I.e. Running that command is unnecessary and does not do anything, because app engine tools are already installed per default, unlike kubectl, which I need. I suspect that if they didn’t do that, the command mentioned would fail.


#8

Just use my instructions above for installing the debian package yourself and setting the path. Works great. Just a bit slow. =(


#9

Thanks all for the responses.

If i try to install kubectl with the debian package it gives me an error message.
Than i’ve tried to install it manualy - that works so far - but “gcloud container clusters get-credentials cluster-2” gives the following error message:

ERROR: (gcloud.container.clusters.get-credentials) The project property is set to the empty string, which is invalid.
To set your project, run:

$ gcloud config set project PROJECT_ID

or to unset it, run:

gcloud container clusters get-credentials cluster-2 returned exit code 1

But the ProjectID is set…

Here are the commands i run:

  - export CLOUD_SDK_REPO="cloud-sdk-$(lsb_release -c -s)" && echo "deb http://packages.cloud.google.com/apt $CLOUD_SDK_REPO main" | sudo tee /etc/apt/sources.list.d/google-cloud-sdk.list
  - curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
  - cat /etc/apt/sources.list.d/google-cloud-sdk.list
  - rm -rf /home/ubuntu/google-cloud-sdk
  - rm -rf /home/ubuntu/.kube
  - export CLOUDSDK_CORE_DISABLE_PROMPTS=1 && curl https://sdk.cloud.google.com | bash
  - gcloud --quiet components update kubectl
  - chmod +x scripts/auth-gcloud.sh
  - chmod +x scripts/rolling-update.sh
  - scripts/auth-gcloud.sh 
  #- gcloud docker push $EXTERNAL_REGISTRY_ENDPOINT/seoviu-wordpress > /dev/null
  - gcloud config set project $GCLOUD_PROJECTID && gcloud config set compute/zone $GCLOUD_ZONE && gcloud config set container/cluster $CLUSTER_NAME
  - gcloud container clusters get-credentials cluster-2

any ideas?


#10

@lookfirst I’m going to try that workarond, but I would prefer if Circle fixed it. :slight_smile:


#11

@MichaelW Are you positive that $GCLOUD_PROJECTID is set? Have you tried echoing it out to verify?


#12

I love the Circle people a lot, but past experience has shown that stuff like this either doesn’t get fixed or takes a very long time. =(


#13

@mpj yes a echo prints the Google Project ID
Local on my mashine the commands work - thats why i’m a litte helpless


#14

As an alternate workaround to installing a fresh Debian package, how about the following? sudo chown ubuntu /opt/google-cloud-sdk -R

I’ve started an internal discussion about this issue–I agree that using kubernetes in deployment should be supported out of the box.


#15

Hi I have tried sudo chown ubuntu /opt/google-cloud-sdk -R then gcloud --quiet components update but failed with saying:

$ yes | gcloud components update


Your current Cloud SDK version is: 101.0.0
You will be upgraded to version: 102.0.0

┌─────────────────────────────────────────────────┐
│        These components will be updated.        │
├──────────────────────────┬────────────┬─────────┤
│           Name           │  Version   │   Size  │
├──────────────────────────┼────────────┼─────────┤
│ Cloud SDK Core Libraries │ 2016.03.22 │ 3.9 MiB │
└──────────────────────────┴────────────┴─────────┘

For the latest full release notes, please visit:
  https://cloud.google.com/sdk/release_notes

Do you want to continue (Y/n)?  
╔════════════════════════════════════════════════════════════╗
╠═ Creating update staging area                             ═╣
ERROR: (gcloud.components.update) Permission denied: [/opt/google-cloud-sdk.staging]

I was using yes here just because the --quiet option seems suspicious to me at that time (it doesn’t actually)

@lookfirst’s solution is working fine for me either.

I am waiting for the fix from CircleCI :smiley:


#16

Thank you for the thorough reply, @tnzk . The engineering team is talking about a more out-of-the-box solution. In the mean time, feel free to use MichaelW’s solution.

Also, another workaround may be to install kubernetes via sudo and then make the logs available for write to all users:

sudo /opt/google-cloud-sdk/bin/gcloud --quiet components update kubectl
sudo chmod 757 /home/ubuntu/.config/gcloud/logs -R

#17

Now that “preview” has been renamed to “beta” in gcloud this is now the case for gcloud -q components update app too.


#18

I’ve got this working per the vanilla guide, but recently gcloud updated something that is is causing
sudo /opt/google-cloud-sdk/bin/gcloud --quiet container clusters get-credentials $GCLOUD_DEV_CLUSTER
to no longer pass credentials off to kubectl correctly and later attempts to apply against my cluster were erroring out with
error: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.

I did a bit of digging around to figure out what was going on, trying out the debian package and ultimately still having the same issue, along with a few new ones (can’t update gcloud components from the dpkg unless you unlock it…). Solved it in the end by pegging my gcloud to version 118.0.0 -

sudo /opt/google-cloud-sdk/bin/gcloud --quiet components update --version 118.0.0
sudo /opt/google-cloud-sdk/bin/gcloud --quiet components update kubectl --version 118.0.0

Hopefully that saves anyone else having a similar issue. Everything else is more or less identical to
https://circleci.com/docs/continuous-deployment-with-google-container-engine/, just split into a few scripts.


#19

We used the fixed version 120.0.0 and it’s worked fine until today when we just got the following:

ERROR: The component listing for Cloud SDK version [120.0.0] could not be found.  Make sure this is a valid archived Cloud SDK version.
ERROR: (gcloud.components.update) Failed to fetch component listing from server. Check your network settings and try again.

It also doesn’t work with 119.0.0 etc.
We also tried to install it with apt-get, but the same error that version cannot be found.

It’s kind of a bummer that we had to revert a version because of the older issue and now this workaround also stopped working. I think it’s time to clean it up CircleCI. cc @Eric

my guesses are:

  • permission issue for the original one (why do we need to run gcloud with sudo?)
  • firewall issue for missing version

Any idea?


#20

Thanks for bringing this to our attention. The issue was fixed in support by clearing the cache for the project. (There were some issues today with connecting to gcloud)