Aws-s3@1.0.4 not working: Unable to install AWS CLI. Please install pip. Error

Ecosystem Orbs
, ,
1

Hello there,

I am trying to deploy a React app in AWS S3 using the sync method from circleci/aws-s3@1.0.4. This is my config.yml code:

version: 2.1
orbs:
  aws-s3: circleci/aws-s3@1.0.4
jobs:
  build:
    docker:
      - image: circleci/node:8.10
    working_directory: ~/flash-web-front
    steps:
      - checkout
      # Download and cache dependencies
      - restore_cache:
          keys:
            - v1-dependencies-{{ checksum "package.json" }}
            # fallback to using the latest cache if no exact match is found
            - v1-dependencies-
      - run: yarn
      - save_cache:
          paths:
            - node_modules
          key: v1-dependencies-{{ checksum "package.json" }}

  # run eslint!
  eslint:
    docker:
      - image: circleci/node:8.10
    working_directory: ~/flash-web-front
    steps:
      - checkout
      - restore_cache:
          keys:
            - v1-dependencies-{{ checksum "package.json" }}
            # fallback to using the latest cache if no exact match is found
            - v1-dependencies-
      - run: yarn lint

  # deploy all
  deploy:
    docker:
      - image: circleci/node:8.10
      - image: circleci/python:3.4
    working_directory: ~/flash-web-front
    steps:
      - checkout
      - restore_cache:
          keys:
            - v1-dependencies-{{ checksum "package.json" }}
            # fallback to using the latest cache if no exact match is found
            - v1-dependencies-
      - run:
          name: Parcel build
          command: yarn build
      - aws-s3/sync:
          from: ~/flash-web-front/dist
          to: '${AWS_S3_WEB_BUCKET}'

workflows:
  version: 2
  build_lint:
    jobs:
      - build
      - eslint:
          requires:
            - build
      - deploy:
          requires:
            - eslint

And this is what the circleci dashboard log:

#!/bin/bash -eo pipefail
export PIP=$(which pip pip3 | head -1)
if [[ -n $PIP ]]; then
  if which sudo > /dev/null; then
    sudo $PIP install awscli --upgrade
  else
    # This installs the AWS CLI to ~/.local/bin. Make sure that ~/.local/bin is in your $PATH.
    $PIP install aws --upgrade --user
  fi
elif [[ $(which unzip curl | wc -l) -eq 2 ]]; then
  cd
  curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
  unzip awscli-bundle.zip
  if which sudo > /dev/null; then
    sudo ~/awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws
  else
    # This installs the AWS CLI to the default location (~/.local/lib/aws) and create a symbolic link (symlink) at ~/bin/aws. Make sure that ~/bin is in your $PATH.
    awscli-bundle/install -b ~/bin/aws
  fi
  rm -rf awscli-bundle*
  cd -
else
  echo "Unable to install AWS CLI. Please install pip."
  exit 1
fi

Plus:

Archive:  awscli-bundle.zip
  inflating: awscli-bundle/install   
  inflating: awscli-bundle/packages/docutils-0.14.tar.gz  
  inflating: awscli-bundle/packages/python-dateutil-2.6.1.tar.gz  
  inflating: awscli-bundle/packages/s3transfer-0.2.0.tar.gz  
  inflating: awscli-bundle/packages/urllib3-1.24.1.tar.gz  
  inflating: awscli-bundle/packages/argparse-1.2.1.tar.gz  
  inflating: awscli-bundle/packages/six-1.12.0.tar.gz  
  inflating: awscli-bundle/packages/botocore-1.12.109.tar.gz  
  inflating: awscli-bundle/packages/ordereddict-1.1.tar.gz  
  inflating: awscli-bundle/packages/python-dateutil-2.8.0.tar.gz  
  inflating: awscli-bundle/packages/simplejson-3.3.0.tar.gz  
  inflating: awscli-bundle/packages/awscli-1.16.119.tar.gz  
  inflating: awscli-bundle/packages/colorama-0.3.9.tar.gz  
  inflating: awscli-bundle/packages/virtualenv-15.1.0.tar.gz  
  inflating: awscli-bundle/packages/futures-3.2.0.tar.gz  
  inflating: awscli-bundle/packages/pyasn1-0.4.5.tar.gz  
  inflating: awscli-bundle/packages/urllib3-1.22.tar.gz  
  inflating: awscli-bundle/packages/jmespath-0.9.4.tar.gz  
  inflating: awscli-bundle/packages/rsa-3.4.2.tar.gz  
  inflating: awscli-bundle/packages/PyYAML-3.13.tar.gz  
  inflating: awscli-bundle/packages/setup/setuptools_scm-1.15.7.tar.gz  
Running cmd: /usr/bin/python virtualenv.py --no-download --python /usr/bin/python /usr/local/aws
Running cmd: /usr/local/aws/bin/pip install --no-cache-dir --no-index --find-links file:///home/circleci/awscli-bundle/packages/setup setuptools_scm-1.15.7.tar.gz
Running cmd: /usr/local/aws/bin/pip install --no-cache-dir --no-index --find-links file:///home/circleci/awscli-bundle/packages awscli-1.16.119.tar.gz
Traceback (most recent call last):
  File "/home/circleci/awscli-bundle/install", line 162, in <module>
    main()
  File "/home/circleci/awscli-bundle/install", line 151, in main
    pip_install_packages(opts.install_dir)
  File "/home/circleci/awscli-bundle/install", line 119, in pip_install_packages
    pip_script, PACKAGES_DIR, cli_tarball))
  File "/home/circleci/awscli-bundle/install", line 49, in run
    p.returncode, cmd, stdout + stderr))
__main__.BadRCError: Bad rc (1) for cmd '/usr/local/aws/bin/pip install --no-cache-dir --no-index --find-links file:///home/circleci/awscli-bundle/packages awscli-1.16.119.tar.gz': Processing ./awscli-1.16.119.tar.gz
Collecting botocore==1.12.109 (from awscli==1.16.119)
Collecting colorama<=0.3.9,>=0.2.5 (from awscli==1.16.119)
Collecting docutils>=0.10 (from awscli==1.16.119)
Collecting rsa<=3.5.0,>=3.1.2 (from awscli==1.16.119)
Collecting s3transfer<0.3.0,>=0.2.0 (from awscli==1.16.119)
Collecting PyYAML<=3.13,>=3.10 (from awscli==1.16.119)
Collecting jmespath<1.0.0,>=0.7.1 (from botocore==1.12.109->awscli==1.16.119)
Collecting python-dateutil<3.0.0,>=2.1 (from botocore==1.12.109->awscli==1.16.119)
Collecting urllib3<1.25,>=1.20 (from botocore==1.12.109->awscli==1.16.119)
Collecting pyasn1>=0.1.3 (from rsa<=3.5.0,>=3.1.2->awscli==1.16.119)
Collecting futures<4.0.0,>=2.2.0 (from s3transfer<0.3.0,>=0.2.0->awscli==1.16.119)
Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore==1.12.109->awscli==1.16.119)
Installing collected packages: jmespath, docutils, six, python-dateutil, urllib3, botocore, colorama, pyasn1, rsa, futures, s3transfer, PyYAML, awscli
  Running setup.py install for jmespath: started
    Running setup.py install for jmespath: finished with status 'done'
  Running setup.py install for docutils: started
    Running setup.py install for docutils: finished with status 'done'
  Running setup.py install for six: started
    Running setup.py install for six: finished with status 'done'
  Running setup.py install for python-dateutil: started
    Running setup.py install for python-dateutil: finished with status 'done'
  Running setup.py install for urllib3: started
    Running setup.py install for urllib3: finished with status 'done'
  Running setup.py install for botocore: started
    Running setup.py install for botocore: finished with status 'done'
  Running setup.py install for colorama: started
    Running setup.py install for colorama: finished with status 'done'
  Running setup.py install for pyasn1: started
    Running setup.py install for pyasn1: finished with status 'done'
  Running setup.py install for rsa: started
    Running setup.py install for rsa: finished with status 'done'
  Running setup.py install for futures: started
    Running setup.py install for futures: finished with status 'done'
  Running setup.py install for s3transfer: started
    Running setup.py install for s3transfer: finished with status 'done'
  Running setup.py install for PyYAML: started
    Running setup.py install for PyYAML: finished with status 'error'
    Complete output from command /usr/local/aws/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-JWA57n/PyYAML/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-_bOZui-record/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/aws/include/site/python2.7/PyYAML:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-2.7
    creating build/lib.linux-x86_64-2.7/yaml
    copying lib/yaml/dumper.py -> build/lib.linux-x86_64-2.7/yaml
    copying lib/yaml/loader.py -> build/lib.linux-x86_64-2.7/yaml
    copying lib/yaml/composer.py -> build/lib.linux-x86_64-2.7/yaml
    copying lib/yaml/__init__.py -> build/lib.linux-x86_64-2.7/yaml
    copying lib/yaml/resolver.py -> build/lib.linux-x86_64-2.7/yaml
    copying lib/yaml/emitter.py -> build/lib.linux-x86_64-2.7/yaml
    copying lib/yaml/nodes.py -> build/lib.linux-x86_64-2.7/yaml
    copying lib/yaml/parser.py -> build/lib.linux-x86_64-2.7/yaml
    copying lib/yaml/events.py -> build/lib.linux-x86_64-2.7/yaml
    copying lib/yaml/cyaml.py -> build/lib.linux-x86_64-2.7/yaml
    copying lib/yaml/constructor.py -> build/lib.linux-x86_64-2.7/yaml
    copying lib/yaml/serializer.py -> build/lib.linux-x86_64-2.7/yaml
    copying lib/yaml/error.py -> build/lib.linux-x86_64-2.7/yaml
    copying lib/yaml/reader.py -> build/lib.linux-x86_64-2.7/yaml
    copying lib/yaml/representer.py -> build/lib.linux-x86_64-2.7/yaml
    copying lib/yaml/tokens.py -> build/lib.linux-x86_64-2.7/yaml
    copying lib/yaml/scanner.py -> build/lib.linux-x86_64-2.7/yaml
    running build_ext
    creating build/temp.linux-x86_64-2.7
    checking if libyaml is compilable
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c build/temp.linux-x86_64-2.7/check_libyaml.c -o build/temp.linux-x86_64-2.7/check_libyaml.o
    checking if libyaml is linkable
    x86_64-linux-gnu-gcc -pthread build/temp.linux-x86_64-2.7/check_libyaml.o -lyaml -o build/temp.linux-x86_64-2.7/check_libyaml
    building '_yaml' extension
    creating build/temp.linux-x86_64-2.7/ext
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c ext/_yaml.c -o build/temp.linux-x86_64-2.7/ext/_yaml.o
    ext/_yaml.c:4:20: fatal error: Python.h: No such file or directory
     #include "Python.h"
                        ^
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    
    ----------------------------------------
Command "/usr/local/aws/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-JWA57n/PyYAML/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-_bOZui-record/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/aws/include/site/python2.7/PyYAML" failed with error code 1 in /tmp/pip-build-JWA57n/PyYAML/

Exited with code 1

Does anyone know what to do here? I believe the circleci/aws-s3 is already up to date.

Thanks!

1 Like
3

Hello,

Installing the AWS CLI requires python, of which pip is a part of. You can install python as part of your job, but it would be better to create a custom image that extends the node image and has python installed. This docs article https://circleci.com/docs/2.0/custom-images/ shows you how to do this.

1 Like
4

Or use one of our circleci/python:node-browsers images:

https://hub.docker.com/r/circleci/python/tags

1 Like
6

Thanks for the answer!

However this doesn’t seem such a fast task hehe. If I install both images (node and python) don’t I get the benefits from both in the same container?

What is actually wrong in my config.yml?

7

Thanks for the answer!

How can check what is available in each Tag? Sorry, I’ve been using circleci for less than 3 days now hehe.

8

No. You can only run commands in your primary container, the first one listed. Docker images can’t be combined into a single container.

I suggest using the image Rose mentioned. You can view some of the tags here and all of the tags in the Python section of this JSON file.

1 Like
9

If you are still encountering issues, can you open an issue under the orb’s repo here https://github.com/CircleCI-Public/circleci-orbs/tree/staging/src/aws-s3 ?

10

Thanks, your answer has opened my head!

I changed the image and now the python related parts are working fine.

It has deployed correctly now, but I have one PS though: you have to set the environment variable AWS_REGION manually, otherwise the aws configure method will crash like this:

#!/bin/bash -eo pipefail
aws configure set region $AWS_REGION \
--profile default
usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters]
To see help text, you can run:

  aws help
  aws <command> help
  aws <command> <subcommand> help
aws: error: the following arguments are required: value
Exited with code 2

I believe there should be a default value for AWS_REGION in case the user hasn’t defined it.

Anyways, thanks to everyone here!

1 Like
11

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.