Psycogp2(SQLAlchemy) cannot find path to files inside the test directory

I am trying to set up CircleCI for my project and it has been pretty frustrating :confounded: .
In my app folder, I have a tests/fixtures directory that contains .txt and .csv files I am using in my tests like so:

class DataBaseTestCases(unittest.TestCase):
    columns = ['method', 'path', 'ip', 'timestamp', 'uuid']

    def setup(self):
        db_conn = DataBaseConnection()
        db_conn.drop_tables()
        db_conn.build_db_if_needed()
        return db_conn

    def csv_upload(self):
        db_conn = self.setup()
        path_to_csv = os.path.abspath(os.path.dirname(__file__))
        with open(os.path.join(path_to_csv,"fixtures/test_csv.csv"), 'w') as out:
            writer = csv.writer(out, delimiter=',')
            writer.writerow(self.columns)
            writer.writerow(['GET',
                             '/',
                             '127.0.0.1',
                             '2017-04-10 17:45:22',
                             'damsjopiwpd'
                            ])

        db_conn.load_csv('pageviews',
                         'test_csv.csv',
                         "{}/fixtures/test_csv.csv".format(os.path.dirname(os.path.realpath(__file__))),
                         item2,
                         'item3',
                         'item4')

The directory structure on my local machine and the CircleCI container is something like this

home/circleci/appfolder/tests
                              test_csv_parser.py
                              test_txt_parser.py
                              test3.py
                               fixtures/
                                           test_file.csv
                                           test_file.txt

I’ve sshed multiple times into the CircleCI container and verified that these files do actually exist, but when the CircleCI build runs, I am getting this error

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not open file "/home/circleci/appfolder/tests/fixtures/test_csv.csv" for reading: No such file or directory

this is my .circleci/config.yml

version: 2
machine:
  services:
    - postgres

jobs:
  build:
    working_directory: ~/appfolder
    docker:
      # Specify the Python version you desire here
      # TODO: figure out a better way to setup postgres see below 
      # https://discuss.circleci.com/t/rails-app-cant-connect-to-postgres-server/13059/3
      - image: circleci/python:3.6.4
        environment:
          PGHOST: 127.0.0.1
          PGUSER: circle
          TEST_DATABASE_URL: postgresql://circle@localhost/dev?sslmode=disable
      - image: circleci/postgres:9.5-alpine
        environment:
          POSTGRES_USER: circle
          POSTGRES_DB: dev
          POSTGRES_PASSWORD: ""

    steps:
      - checkout

      - run:
         name: create virtualenv and install python dependencies
         command: |
           python3 -m venv venv
           . venv/bin/activate
           pip install -r requirements.txt

      - run:
         name: Wait for DB
         command: dockerize -wait tcp://localhost:5432 -timeout 1m

      - run: 
         name: install postgres-client(psql)
         command: sudo apt-get install -y postgresql-client

      - run:
          name: tests
          command: |
            . venv/bin/activate
            python tests/test.py

      - run:
         name: coverage report
         command: |
          . venv/bin/activate
          py.test --cov=src tests/

What am I doing wrong? why is CircleCI not finding the test fixtures files even though they do exist?

When you SSH in, what happens if you run the tests at that point?

I get the same error as above.

OK, that’s a debuggable problem then! Try:

  • Hardwiring the actual path temporarily (remove os.path.join(path_to_csv and format(os.path.dirname(os.path.realpath(__file__))))
  • Doing a Python file exists test on the file (path hardwired again)
  • Doing an ls -R /home/circleci/appfolder/tests command using a Python system call

I’d be looking for string case mismatches, weird control characters, Unix ownership and permission issues, etc.

I tried doing a subprocess

s = subprocess.Popen(["ls", "-R", "/home/circleci/appfolder/tests/fixtures/"], stdout=subprocess.PIPE)
        print(s.communicate()[0])

and got

b'/home/circleci/appfolder/tests/fixtures/:\ntest_csv.csv\ntest_event_log.txt\ntest_pageview_log.txt\n'
Eb'/home/circleci/appfolder/tests/fixtures/:\ntest_csv.csv\ntest_event_log.txt\ntest_pageview_log.txt\n'

I am not familiar with hardwiring or how to do that so any link to docs or blog is appreciated.

So far, It seems all the files do exist.

IMO, “hardwiring” is not a thing one learns about it books and blogs. It’s just an informal name for a hacking technique - in means “doing technical work in a rough and unfinished fashion”. For example, one should never put in home directory paths (/home/circleci/) since that hardwires your code to that directory (which is not very flexible). But for debugging, it is a useful technique - maybe your Python functions to determine the current directory are playing up.

Also, if you can push a minimum working example to a public Git repo, that can help - presently no reader versed in Python can easily run this for themselves. Make it so you try to tempt someone to help on the Circle platform in a few minutes.

I would encourage you also not to see this as a Circle problem - it is likely to be either a Python problem or an environmental problem. I’m afraid I don’t know Python though, so nothing jumps out at me. If you have reached the limit of your knowledge, ask a question on Stack Overflow or other good Q&A site.

I didn’t understand your console output, perhaps because I don’t know Python. What did that show, in plain English?