Psycogp2(SQLAlchemy) cannot find path to files inside the test directory

Build Environment Databases
1

I am trying to set up CircleCI for my project and it has been pretty frustrating :confounded: .
In my app folder, I have a tests/fixtures directory that contains .txt and .csv files I am using in my tests like so:

class DataBaseTestCases(unittest.TestCase):
    columns = ['method', 'path', 'ip', 'timestamp', 'uuid']

    def setup(self):
        db_conn = DataBaseConnection()
        db_conn.drop_tables()
        db_conn.build_db_if_needed()
        return db_conn

    def csv_upload(self):
        db_conn = self.setup()
        path_to_csv = os.path.abspath(os.path.dirname(__file__))
        with open(os.path.join(path_to_csv,"fixtures/test_csv.csv"), 'w') as out:
            writer = csv.writer(out, delimiter=',')
            writer.writerow(self.columns)
            writer.writerow(['GET',
                             '/',
                             '127.0.0.1',
                             '2017-04-10 17:45:22',
                             'damsjopiwpd'
                            ])

        db_conn.load_csv('pageviews',
                         'test_csv.csv',
                         "{}/fixtures/test_csv.csv".format(os.path.dirname(os.path.realpath(__file__))),
                         item2,
                         'item3',
                         'item4')

The directory structure on my local machine and the CircleCI container is something like this

home/circleci/appfolder/tests
                              test_csv_parser.py
                              test_txt_parser.py
                              test3.py
                               fixtures/
                                           test_file.csv
                                           test_file.txt

Iā€™ve sshed multiple times into the CircleCI container and verified that these files do actually exist, but when the CircleCI build runs, I am getting this error

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not open file "/home/circleci/appfolder/tests/fixtures/test_csv.csv" for reading: No such file or directory

this is my .circleci/config.yml

version: 2
machine:
  services:
    - postgres

jobs:
  build:
    working_directory: ~/appfolder
    docker:
      # Specify the Python version you desire here
      # TODO: figure out a better way to setup postgres see below 
      # https://discuss.circleci.com/t/rails-app-cant-connect-to-postgres-server/13059/3
      - image: circleci/python:3.6.4
        environment:
          PGHOST: 127.0.0.1
          PGUSER: circle
          TEST_DATABASE_URL: postgresql://circle@localhost/dev?sslmode=disable
      - image: circleci/postgres:9.5-alpine
        environment:
          POSTGRES_USER: circle
          POSTGRES_DB: dev
          POSTGRES_PASSWORD: ""

    steps:
      - checkout

      - run:
         name: create virtualenv and install python dependencies
         command: |
           python3 -m venv venv
           . venv/bin/activate
           pip install -r requirements.txt

      - run:
         name: Wait for DB
         command: dockerize -wait tcp://localhost:5432 -timeout 1m

      - run: 
         name: install postgres-client(psql)
         command: sudo apt-get install -y postgresql-client

      - run:
          name: tests
          command: |
            . venv/bin/activate
            python tests/test.py

      - run:
         name: coverage report
         command: |
          . venv/bin/activate
          py.test --cov=src tests/

What am I doing wrong? why is CircleCI not finding the test fixtures files even though they do exist?

2

When you SSH in, what happens if you run the tests at that point?

3

I get the same error as above.

4

OK, thatā€™s a debuggable problem then! Try:

  • Hardwiring the actual path temporarily (remove os.path.join(path_to_csv and format(os.path.dirname(os.path.realpath(__file__))))
  • Doing a Python file exists test on the file (path hardwired again)
  • Doing an ls -R /home/circleci/appfolder/tests command using a Python system call

Iā€™d be looking for string case mismatches, weird control characters, Unix ownership and permission issues, etc.

5

I tried doing a subprocess

s = subprocess.Popen(["ls", "-R", "/home/circleci/appfolder/tests/fixtures/"], stdout=subprocess.PIPE)
        print(s.communicate()[0])

and got

b'/home/circleci/appfolder/tests/fixtures/:\ntest_csv.csv\ntest_event_log.txt\ntest_pageview_log.txt\n'
Eb'/home/circleci/appfolder/tests/fixtures/:\ntest_csv.csv\ntest_event_log.txt\ntest_pageview_log.txt\n'

I am not familiar with hardwiring or how to do that so any link to docs or blog is appreciated.

So far, It seems all the files do exist.

6

IMO, ā€œhardwiringā€ is not a thing one learns about it books and blogs. Itā€™s just an informal name for a hacking technique - in means ā€œdoing technical work in a rough and unfinished fashionā€. For example, one should never put in home directory paths (/home/circleci/) since that hardwires your code to that directory (which is not very flexible). But for debugging, it is a useful technique - maybe your Python functions to determine the current directory are playing up.

Also, if you can push a minimum working example to a public Git repo, that can help - presently no reader versed in Python can easily run this for themselves. Make it so you try to tempt someone to help on the Circle platform in a few minutes.

I would encourage you also not to see this as a Circle problem - it is likely to be either a Python problem or an environmental problem. Iā€™m afraid I donā€™t know Python though, so nothing jumps out at me. If you have reached the limit of your knowledge, ask a question on Stack Overflow or other good Q&A site.

I didnā€™t understand your console output, perhaps because I donā€™t know Python. What did that show, in plain English?