Unable to migrate circleci from 1.0 To 2.0

circle.yml
#1

Hello Everyone,
I have been trying t fix the issues with cci 2.0 since last 7 - 10 days, but no luck. I have tried multiple ways. But failing with php commands under steps. It would be really great if somebody can help with writing 2.0 config.yml file. Appreciate your help.

circleci 1.0

version: 1
machine:
php:
version: 7.1.9
environment:
APP_ENV: testing
APP_KEY: asdsdasdfdfasdfadf
DB_HOST: 127.0.0.1
DB_DATABASE: db_name
DB_USERNAME: username
image: selenium/standalone-chrome:3.1.0

general:
artifacts:
- “reports”

dependencies:
pre:
- printf “\n” | pecl install -f imagick
- echo “extension = imagick.so” >> /opt/circleci/php/$(phpenv global)/etc/php.ini
override:
# Checks all php files within /app for syntax errors
- tests/scripts/phplint.sh
- composer install --prefer-dist --no-interaction --no-dev
- ‘php artisan migrate --no-interaction --seed’
# Run the CS fixer and error if it finds files to update
- tests/scripts/phpcs.sh
cache_directories:
- “~/.composer/cache”
- “reports/phplint/.cache”
- “vendor”

test:
override:
- vendor/bin/phpunit -d memory_limit=1024M --coverage-clover build/logs/clover.xml
- vendor/bin/behat

deployment:
production:
branch: production
commands:
- rsync -rl . ‘–usermap=*:www-data’ --exclude=’.env’ --exclude=‘build’ "xyz@123.primarydomain.com:/var/www/ici.primarydomain.com/"
staging:
branch: master
commands:
- rsync -rl . ‘–usermap=*:www-data’ --exclude=’.env’ --exclude=‘build’ ‘123@boethius.primarydomain.com:/var/www/mykw/’

circleci 2.0
version: 2

jobs:
build:
docker:
- image: php:7.1.9
environment:
APP_ENV: testing
APP_KEY: base64:ADFASDFADADFD
DB_HOST: 127.0.0.1
DB_DATABASE: db_name
DB_USERNAME: username
- image: selenium/standalone-chrome:3.1.0

working_directory: /var/www/mykw

steps:
  - checkout
  # Checks all php files within /app for syntax errors
  - run: tests/scripts/phplint.sh
  - run: php -derror_reporting=-1 -ddisplay_errors=1 -r "copy('https://getcomposer.org/installer', 'composer-setup.php');"
  - run: php -r "if (hash_file('sha384', 'composer-setup.php') === '48e3236262b34d30969dca3c37281b3b4bbe3221bda826ac6a9a62d6444cdb0dcd0615698a5cbe587c3f0fe57a54d8f5') { echo 'Installer verified'; } else { echo 'Installer corrupt'; unlink('composer-setup.php'); } echo PHP_EOL;"
  - run: php composer-setup.php
  - run: php -r "unlink('composer-setup.php');"
  - run: php composer.phar self-update
  # - run: sudo mv composer.phar /usr/local/bin/composer
  # - run: composer install -n --prefer-dist --no-interaction --no-dev
  # - run: php artisan migrate --no-interaction --seed
  # Run the CS fixer and error if it finds files to update
  - run: tests/scripts/phpcs.sh
  - run: rsync -rl . '--usermap=\*:www-data' --exclude='.env' --exclude='build' 'deploy@boethius.threesolve.com:/var/www/mykw/'

general:
artifacts:
- “reports”

dependencies:
pre:
- printf “\n” | pecl install -f imagick
- echo “extension = imagick.so” >> /opt/circleci/php/$(phpenv global)/etc/php.ini
override:
# Checks all php files within /app for syntax errors
- tests/scripts/phplint.sh
- composer install --prefer-dist --no-interaction --no-dev
- ‘php artisan migrate --no-interaction --seed’
# Run the CS fixer and error if it finds files to update
- tests/scripts/phpcs.sh
cache_directories:
- “~/.composer/cache”
- “reports/phplint/.cache”
- “vendor”

test:
override:
- vendor/bin/phpunit -d memory_limit=1024M --coverage-clover build/logs/clover.xml
- vendor/bin/behat

deployment:
production:
branch: production
commands:
- rsync -rl . ‘–usermap=*:www-data’ --exclude=’.env’ --exclude=‘build’ "xyz@123.primarydomain.com:/var/www/ici.primarydomain.com/"
staging:
branch: master
commands:
- rsync -rl . ‘–usermap=*:www-data’ --exclude=’.env’ --exclude=‘build’ ‘123@boethius.primarydomain.com:/var/www/mykw/’

#2

Before people can assist you, your YAML needs to be readable. There are code formatting tools in this forum - would you apply code formatting to each piece of config? Markdown is supported. To format code here, use this syntax:

```
Your file
```

Once this is done, readers will need to see how the step is failing. A detailed report of what happens (e.g. an error) and what you expected to happen (e.g. some tests were run) is essential for you to get useful help.

In general, detailed questions are much more answerable than vague ones, and thus detailed questions are much more likely to get an answer. For more information on this, see this excellent advice. Although it was written with another site in mind, it generalises very well to any source of assistance on the internet.

#3

Thank you for your suggestion. But i’m not able to any formats for code snippet. I tried alot and spent some time to understand it. But no luck.

defaults: &defaults
  working_directory: /var/www/mykw
  docker:
    - image: circleci/php:7.1.9
version: 2
jobs:
  build:
    <<: *defaults
environment:
  APP_ENV: testing
  APP_KEY: base64:nXcpXrXevvAuvJLCpXQc8rMSM9OGz8hG0G4Jf3SXsgo=
  APP_DEBUG: true
  APP_LOG_LEVEL: debug
  NODE_ENV: testing
  DB_HOST: 127.0.0.1
  DB_DATABASE: circle_test
  DB_USERNAME: ubuntu
steps:
  - checkout
  
  # Download and cache dependencies
  - restore_cache:
    keys:
    - v1-dependencies-{{ checksum "composer.json" }}
    # fallback to using the latest cache if no exact match is found
    - v1-dependencies-
  - run: composer install -n --prefer-dist
  
  -save_cache:
    paths:
      - ./vendor
    key: v1-dependencies-{{ checksum "composer.json" }}
	
  # Run Tests !
  - run: vendor/bin/phpunit -d memory_limit=1024M --coverage-clover build/logs/clover.xml
  - run: vendor/bin/behat

  deploy_master:
<<: *defaults
steps:
  - run:
      name: Deploy Master Branch on Stage
      command: - rsync -rl . '--usermap=\*:www-data' --exclude='.env' --exclude='build' 'deploy@boethius.threesolve.com:/var/www/mykw/'

workflows:
  version: 2
  notify_deploy:
jobs:
  - build
  - deploy_master:
      requires:
        - build
      filters:
        branches:
          only: master

This is the script file i have written for CCI 2.0 but we are getting errors as following.
2 schema violations found
required key [jobs] not found
required key [version] not found

#4

@halfer - Could you please go through the above formatted if you get any chance.?

Friends, it would be very grateful if anybody can help to resolve this issue.

Thanks in Advance

#5

Hi @venukommu. This section appears at the wrong indent levels. Is that a bad copy-paste, or is that the case? I think that will fix the jobs error, and might help the version one as well.

You can use the CLI to help with errors like this https://circleci.com/docs/2.0/local-cli/#validate-a-circleci-config

#6

Hi @drazisil, thank you for taking time to look into it. I have verified this script through YML validator and shown nothing wrong. I have tried by updating the script as you suggested, but we are getting the same issues. I took some working yml files from git and updated accordingly, there also we are getting same issues.

We are not able to understand the actual issue that either it is cci 2.0 or format of the file.

Appreciate if any suggestions.

#7

Hi @drazisil @halfer,
I have changed the script completely and updated as following…

  version: 2
  jobs:
  build:
  docker:

  # specify the version you desire here
  - image: circleci/php:7.3.3

  environment:
  APP_ENV: testing
  APP_KEY: base64:nXcpXrXevvAuvJLCpXQc8rMSM9OGz8hG0G4Jf3SXsgo=
  DB_HOST: 127.0.0.1
  DB_DATABASE: circle_test
  DB_USERNAME: ubuntu

  - image: selenium/standalone-chrome:3.1.0

  # Specify service dependencies here if necessary
  # CircleCI maintains a library of pre-built images
  # documented at https://circleci.com/docs/2.0/circleci-images/
  - image: circleci/mysql:5.7.18
    environment:
      MYSQL_DATABASE: homestead
      MYSQL_USER: homestead
      MYSQL_PASSWORD: secret
      MYSQL_ROOT_PASSWORD: secret
      #DB_CONNECTION=mysql
      #DB_HOST=127.0.0.1
      #DB_PORT=3306
      #DB_DATABASE=homestead
      #DB_USERNAME=homestead
      #DB_PASSWORD=secret

working_directory: /var/www/mykw/

steps:
  - checkout
  - run: printf "\n" | pecl install -f imagick
  - run: echo "extension = imagick.so" >> /opt/circleci/php/$(phpenv global)/etc/php.ini
  - run: chmod -R 777 storage
  - run: cp .env.example .env
  # Checks all php files within /app for syntax errors
  - run: tests/scripts/phplint.sh  

  # Download and cache dependencies
  # Composer
  - restore_cache:
      name: Restore composer cache
      keys:
      - composer-{{ checksum "composer.json" }}
      - composer-

  # Checks all php files within /app for syntax errors
  - run: php artisan optimize
  - run: composer install --prefer-dist --no-interaction --no-dev
  - run: 'php artisan migrate --no-interaction --seed'

  # Run the CS fixer and error if it finds files to update
  - run: tests/scripts/phpcs.sh
  - save_cache:
      name: Store composer cache
      key: composer-{{ checksum "composer.json" }}
      paths:
        - vendor
  # Yarn
  - restore_cache:
      name: Restore yarn cache
      keys:
        - yarn-{{ checksum "yarn.lock" }}
        - yarn-
  - run: yarn
  - save_cache:
      paths:
        - node_modules
      key: yarn-{{ checksum "composer.json" }}

  - run: npm run master
  - run: php artisan migrate --database=homestead

  # mkdir -p /phpunit &&
  # php -dzend_extension=xdebug.so vendor/bin/phpunit --coverage-clover=phpunit/coverage-clover.xml --coverage-html=phpunit/coverage
  # run tests!
  - run:
      command: |
        - rsync -rl . '--usermap=\*:www-data' --exclude='.env' --exclude='build' 'deploy@boethius.threesolve.com:/var/www/mykw/'

  - store_test_results:
      path: /phpunit
  - store_artifacts:
      path: /phpunit

Actually it is working fine without configuration issues, but here we are getting following issues with all commands that we tried to run with "- run: ".

Issue we are getting for almost all commands:
#!/bin/bash -eo pipefail - rsync -rl . ‘–usermap=*:www-data’ --exclude=’.env’ --exclude=‘build’ ‘deploy@boethius.threesolve.com:/var/www/mykw/’

Unable to create directory ‘/var/www/mykw’: mkdir /var/www/mykw: permission denied

Already there was directory called " /var/www/mykw", but again it is trying to create another directory.

Appreciate if any body can help me with the issue.

#8

You are telling all commands to run in a directory that doesn’t exist on the build machine, and that you don’t have permission to create without sudo. You should run all your steps in the default working directory, then create that dir and cd into it for the steps where you need to.

#9

Hi @drazisil, thank you for your reply. Here you can check the attached screen. Mentioned path in the config file is available on server.

Appreciate your help and prompt response.
Thank You
Venu Kommu

#10

I have changed working directory as following

working_directory: ~/var/www/mykw/

then we are able to checkout the code without issues, but the problem is, it is checking out the code in /home/circleci/var/www/mykw/ instead of /var/www/mykw/ directory which is not existed. But we need to checkout and isntall everything in /var/www/mykw/ directory.

Can anyone suggest how to mention the path ?

If we give path like /var/www/mykw/ then it was saying no permissions to create directory, if we mention ~ before path (~/var/www/mykw/) then it is creating same path in /home/circleci.

Appreciate if anybody can help us.

#11

I would recommend against that, for two reasons:

  • it feels like you are fighting the CI rather than working with it - just use the default working directory, where file permissions and ownership are set correctly
  • your app and tests should not be dependent on where they are stored, and you should consider them brittle until that is fixed

To be clear, you probably can get it working in /var/www, but as I say, I would change course if you can.

#12

@halfer - may be you are right, because of struggling with CCI 2.0 since two weeks. :slight_smile: So it might effect my way of thinking… We are getting following issue if we mention working directory as /var/www/mykw or /var/www/.

#!/bin/sh
set -e
# Workaround old docker images with incorrect $HOME
# check https://github.com/docker/docker/issues/2968 for details
if [ "${HOME}" = "/" ]
then
  export HOME=$(getent passwd $(id -un) | cut -d: -f6)
fi
mkdir -p ~/.ssh

 echo 'github.com ssh-rsa     AAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==
bitbucket.org ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAubiN81eDcafrgMeLzaFPsw2kNvEcqTKl/VqLat/MaB33pZy0y3rJZtnqwR2qOOvbwKZYKiEO1O6VqNEBxKvJJelCq0dTXWT5pbO2gDXC6h6QDXCaHo6pOHGPUy+YBaGQRGuSusMEASYiWunYN0vCAI8QaXnWMXNMdFP3jHAJH0eDsoiGnLPBlBp4TNm6rYI74nMzgz3B9IikW4WVK+dc8KZJZWYjAuORU3jc1c/NPskD2ASinf8v3xnfXeukU0sJ5N6m5E8VLjObPEO+mN2t/FZTMZLiFqPWc/ALSqnMnnhwrNi2rbfg/rd/IpL8Le3pSBne8+seeFVBoGqzHM9yXw==
' >> ~/.ssh/known_hosts

(umask 077; touch ~/.ssh/id_rsa)
chmod 0600 ~/.ssh/id_rsa
(cat <<EOF > ~/.ssh/id_rsa
$CHECKOUT_KEY
EOF
)

# use git+ssh instead of https
git config --global url."ssh://git@github.com".insteadOf "https://github.com" || true
git config --global gc.auto 0 || true

if [ -e /var/www/.git ]
then
  cd /var/www
  git remote set-url origin "$CIRCLE_REPOSITORY_URL" || true
 else
  mkdir -p /var/www
  cd /var/www
  git clone "$CIRCLE_REPOSITORY_URL" .
fi

if [ -n "$CIRCLE_TAG" ]
then
  git fetch --force origin "refs/tags/${CIRCLE_TAG}"
else
  git fetch --force origin "CCI-1-0-Migration-To-CCI-2-0:remotes/origin/CCI-1-0-Migration-To-CCI-2-0"
fi


if [ -n "$CIRCLE_TAG" ]
then
  git reset --hard "$CIRCLE_SHA1"
  git checkout -q "$CIRCLE_TAG"
elif [ -n "$CIRCLE_BRANCH" ]
then
  git reset --hard "$CIRCLE_SHA1"
  git checkout -q -B "$CIRCLE_BRANCH"
fi

git reset --hard "$CIRCLE_SHA1"
mkdir: cannot create directory ‘/var/www’: Permission denied
 Exited with code 1

Appreciate if any help.

#13

@halfer @drazisil - It was moved forward when i mentioned working directory as ~/var/www/mykw, but
we have another problem here.

> Illuminate\Foundation\ComposerScripts::postAutoloadDump
> php artisan package:discover
In PDOConnection.php line 50:
                         
  could not find driver  
                         

In PDOConnection.php line 46:
                         
  could not find driver  
                     

Script php artisan package:discover handling the post-autoload-dump event returned with error code 1
Exited with code 1

I have verified manually and there is mysql driver installed on the server. Config file is also showing mysql driver as default.

Here you can see the PHP Info
https://ocean.knowledgeworkx.com/phpinfo.php

Appreciate your help.

#14

The phpinfo on your host makes me wonder if I have misunderstood the problem - isn’t the problem that you are trying to get a build working on CircleCI? If so, the extensions enabled on your host won’t affect it - it’s what you have installed in the build server that counts.

It looks like you are using PDO, so make sure you have PDO and a database-specific ext installed, probably pdo_mysql. If either of these are missing then you will need to install them in your CircleCI config. It’s usually apt-get install -y php7-pdo or similar (depending on the names in the upstream repo).

#15

Hi @halfer, yes… It was resolved when i installed PDO driver. But i see lot of challenges to install that driver. I think i have moved little forward and again fighting to fix the following issue.

checking for location of ICU headers and libraries... not found
configure: error: Unable to detect ICU prefix or no failed. Please verify ICU install prefix and make sure icu-config works.
ERROR: `/tmp/pear/temp/intl/configure --with-php-config=/usr/local/bin/php-config --with-icu-dir=DEFAULT' failed
Exited with code 1

I tried to install libicu-dev icu-devtools icu-doc packages, but we are getting some other issues while trying to install it.

E: Failed to fetch http://deb.debian.org/debian/pool/main/i/icu/icu-devtools_52.1-8+deb8u6_amd64.deb  404  Not Found

E: Failed to fetch http://deb.debian.org/debian/pool/main/i/icu/icu-doc_52.1-8+deb8u6_all.deb  404  Not Found

E: Failed to fetch http://deb.debian.org/debian/pool/main/i/icu/libicu-dev_52.1-8+deb8u6_amd64.deb  404  Not Found

Appreciate if anybody can help us to resolve these issues.
Thank you very much

#16

It may be worth my pointing out that the problems you are running into are not CircleCI problems. They are Linux problems, and so can be researched on the wider web (e.g. Stack Overflow).

It looks here like you are trying to compile the PHP “intl” extension, and are missing some operating system dependencies. Something is trying to install those dependencies but is not finding them - maybe you need an apt-get update -y beforehand? It’s hard to say, given that the config provided does not seem to mention this extension.

If you are frequently getting stuck, would some dedicated assistance from a colleague or a consultant be a possibility? It may be cheaper (and quicker) in the long run to pay someone to help you.

#17

Hello @halfer, thank you very much for your suggestion. Already i have researched alot and tried multiple solutions. Actually we don’t wanted to update all these things like Ubuntu, PHP and Laravel. As a first step, we just wanted to migrate cci. But upgraded cci 2.0 is prompting to update all these stuff one by one. Actually the same application has been working well on the same server since 3 to 4 years. But cci 1.0 was expired and it is not allowing us to deploy the newly committed code changes.

If we can deploy the application as it was by using cci 2.0 that would be really great and appreciate you. I’m not able to understand why is it asking to update everything one by one when we are migrating 1.0 to 2.0. We have used the same error and application working fine using 1.0 currently.

Problem is, no one knows about CCI here in our office. I’m the one who has to migrate this kind of services. It might need to think about to take paid service. Could you please help me in that case ?

Appreciate your guidance
Thank You

#18

@halfer @drazisil @djones8520 - Hi, I see some progress here. Appreciate if anyone an help me to debug the issue. Almost i have fixed all the issues that i have received so far, finally i have stopped with DB connection. I’m getting following issue at end. I have tried to connect manually using the same credentials and worked well on Linux server.

Illuminate\Foundation\ComposerScripts::postAutoloadDump
@php artisan package:discover

In PDOConnection.php line 50:
                                                                               
  SQLSTATE[HY000] [1045] Access denied for user 'mykw'@'127.0.0.1' (using pas  
  sword: YES)                                                                  
                                                                             
In PDOConnection.php line 46:
                                                                               
  SQLSTATE[HY000] [1045] Access denied for user 'mykw'@'127.0.0.1' (using pas  
  sword: YES)                                                                  
                                                                           

Script @php artisan package:discover handling the post-autoload-dump event returned with error code 1
Exited with code 1

I was able to connect when i tried to connect with same credentials over the Linux console, not working same credentials through CCI 2.0

Really appreciate if anyone can help us.

#19

Just to update: It is working well when i replaced #DB_DATABASE, #DB_USERNAME, DB_PASSWORD with MYSQL_**** instead of DB_***.

But got another issue with :slight_smile:

In SchemaException.php line 89:
                                                            
  There is no column with name 'role_id' on table 'users'.  

Exited with code 1 

It might be an issue with Tables.

#20

This problem is a PHP issue, and does not have anything to do with CI. Assuming this is still your migration step, either your migrations or your seeds do not run from scratch. I would guess that these work locally because your database is already set up locally, and the migrations/seeds are considered to have run.

If you drop your local database and create an empty one, and then try to run these migrations there, I expect they would fail. That’s your next step - recreating the bug on your local machine.