Unable to open sqlite3 file in build although it works locally

Build Environment Databases
1

Hello,

I use multiple sqlite3 databases in a repo and circle CI does not want get data from them. I am using relative Unix path (using python pkg_resource) so there is no reason it would not work. It fails with both sqlalchemy and sqlite3 libraries. I have resorted to using JSON files instead but this is a bad option for my use case.

EDIT: I may have found the culprit (sorry idk how it did not come to my mind the past months…). I am using git LFS for the sqlite3 file. I thought due to that the file is not there but it is (just used python’s os.walk on Circleci to make sure of it). So I probably “just” need to initialize git LFS in CircleCI (actually it seems very complicated to me from what I’ve read so far).

Code sample and error on Circle CI 
import sqlalchemy
import pkg_resources

# acquire database path relative to my private package path then format it as Unix
database_path = pkg_resources.resource_filename('data_cleansing.geo','geonames.sqlite3')
database_path = database_path.replace('\\','/')
print(f'Opening database at {database_path}')

# create engine to interact with the database
engine = sqlalchemy.create_engine(f'sqlite:///{database_path}')

# run test query and get results
with engine.connect() as con:
    results = con.execute('SELECT DISTINCT(ISO) FROM countries_info').fetchall()
    
print(results)

Traceback (most recent call last):
File “./data_cleansing/all/tests/test_data_completion.py”, line 8, in
from data_cleansing.all import data_completion
File “/home/circleci/data_cleansing/data_cleansing/init.py”, line 7, in
import(module)
File “/home/circleci/data_cleansing/data_cleansing/geo/database_test_circle_ci.py”, line 25, in
results = con.execute(‘SELECT DISTINCT(ISO) FROM countries_info’).fetchall()
File “/home/circleci/data_cleansing/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py”, line 982, in execute
return self.execute_text(object, multiparams, params)
File “/home/circleci/data_cleansing/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py”, line 1155, in _execute_text
parameters,
File “/home/circleci/data_cleansing/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py”, line 1248, in _execute_context
e, statement, parameters, cursor, context
File “/home/circleci/data_cleansing/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py”, line 1466, in _handle_dbapi_exception
util.raise_from_cause(sqlalchemy_exception, exc_info)
File “/home/circleci/data_cleansing/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py”, line 399, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File “/home/circleci/data_cleansing/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py”, line 153, in reraise
raise value.with_traceback(tb)
File “/home/circleci/data_cleansing/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py”, line 1244, in _execute_context
cursor, statement, parameters, context
File “/home/circleci/data_cleansing/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py”, line 550, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.DatabaseError: (sqlite3.DatabaseError) file is encrypted or is not a database
[SQL: SELECT DISTINCT(ISO) FROM countries_info]
(Background on this error at: http://sqlalche.me/e/4xp6)
Exited with code 1

Thanks for your help in advance.

2

Have you tried to ssh into the job and run your steps manually? I would suggest jumping into the job and manually making sure the file is there and intact Then run your steps line by line. I can’t think of why it wouldn’t connect from the info here but I think you’ll discover it pretty fast if you work through each step individually from the command line.

3

Hi fernfernfern,

I will try to do that as soon as possible (I need to fix this by the end of the month but was assigned an important task yesterday). Thanks :slight_smile:

4

Hi fernfernfern,

I just had an idea and I was right. The file is just a placeholder somehow. The size is only 134 bytes. It’s missing some 600 MB or so :stuck_out_tongue:… So yeah I’ll try to configure this LFS thing somehow.

5

soooo I solved the problem by not solving the problem. I used compressed and splitted json files to avoid using git lfs and it does the trick. Not a very elegant solution but it works :slight_smile:.

1 Like