Testing in different environments

Build Environment Programming Languages
,
1

I would like to test a Django package in different Django environments.

Travis CI has the convenient option of defining the Python and Django version which generates the permutations of the tests.

I was looking for a similar setup option for the yaml files in CircleCI, but I can’t find this option in the documentation.

How can I test a package on CircleCI within different environments in the same test run?

2

Tox is a tool for automate testing in Python. It does so by reading
a tox.ini file where we specify the environments we want to test, and it will create a brand new
virtual environment for that setup and run the test suite against it.
Let’s start by installing tox and add it to our requirements.txt file.

(django-foobar):~/django-foobar$ pip install tox
(django-foobar):~/django-foobar$ echo “tox” >> requirements.txt

Create a new file tox.ini in the projects working directory and paste the content below.

[tox]
envlist =
py27-django1.6,
py34-django1.8

[django1.6]
deps =
Django>=1.6,<1.7

[django1.8]
deps =
Django>=1.8,<1.9

[testenv]
commands =
python {toxinidir}/setup.py test

[testenv:py27-django1.6]
basepython = python2.7
deps =
{[django1.6]deps}

[testenv:py34-django1.8]
basepython = python3.4
deps =
{[django1.8]deps}

As you can see we have a [tox] block in which we defines a list of environments we want to test.
Next, we define some blocks for the different Django versions we want to test. The [testenv] block
defines the command we want to run, and finally we combine the [testenv] with the different version
blocks in order to fire up the test runner.

3

One option would be to to use tox if you would like to run the test sequentially on two (or more) Python versions. Some customers have reported success with the following configuration:

tox.ini

[tox]
envlist = py26,py34
[testenv]
commands=pip install -e .
         pip install -r requirements-dev.txt 
         python setup.py test 
install_command=pip install --process-dependency-links --allow-external --allow-unverified {opts} {packages}

circle.yml

dependencies:
   pre:
     - pip install tox

test:
   override:
     - tox

Another option would be to simply change the Python version, either during the build or on one of the containers in a parallel build. We use pyenv to manage Python versions, so you could do something like this if you are using a single container:

machine:
  python:
    version: 2.6.8
​
test:
  override:
    - make test
    - pyenv global 2.7.9
    - make test

Alternatively, you can test both versions in parallel—e.g. set up a parallel build and make a few build containers run different versions of Python.

We add an env var called $CIRCLE_NODE_INDEX when a build runs with more than one container. This means that you can just make the Python choice with a simple if condition in the dependencies: pre step. Mind that the machine section cannot handle multiple configurations, so you could do something like this:

machine:
  python:
    version: 2.6.8
​
dependencies:
  pre:
    - if [ $CIRCLE_NODE_INDEX == "1" ] ; then pyenv global 2.7.9 ; fi
​
test:
  override:
    - make test: # note the colon
        parallel: true

​
This will configure Python 2.7.9 instead of 2.6.8 only on the second container (index starts at 0).

1 Like
4

Thank you for your replies! Great community!

5

Using this method for 3.5.0 + py.test and it didn’t seem to change which version of Python was used?

Thanks for any help, appreciate it!

6

Could you please post the exact steps you have in your circle.yml? I will try to reproduce it and figure out what could have gone wrong. Thanks.

7 
machine:
  python:
    version: 2.7.5

dependencies:
  override:
    - pip install -r requirements.txt
test:
  override:
    - pyenv global 3.5.0
    - py.test

With that, py.test ran with 2.7.5

Repo here: https://github.com/ckcollab/cassiopeia

Tests here: https://circleci.com/gh/ckcollab/cassiopeia

8

I was able to reproduce this—looks like the reason is the virtualenv that we create by default to ensure the consistency of the Python environment.

You could do something like this to work around it:

test:
  override:
    - deactivate; pyenv global 3.5.0; py.test

Could you please try that out?

9

I got…

deactivate; pyenv global 3.5.0; py.test

bash: line 1: py.test: command not found deactivate; pyenv global 3.5.0; py.test returned exit code 127

Nothing seems to be working with pyenv global 3.5.0, tried these as well (no deactivate):

pyenv global 3.5.0; py.test
pyenv global 3.5.0 && py.test
# these actually run, but as 2.7.5

ALSO, side problem, Python 2.7 on Circle seems to not work with urllib + https? See this thread: Python 2 not "built with ssl module"?

10

First of all, thanks for the detailed reply, that is a nice workaround.
But I wouldnt call it a solution. It is a terrible hack.
Are there any plans, to support this scenario directly from circle.yml?
While for a specific end-product this is a rare scenario, a FOSS usually requires testing against different configurations.
Thank you,
vhermecz

11

I struggled with this a little bit today, but ended up with a solution that seems not so bad after all. I did a quick write up of it here . . .

https://ben.fogbutter.com/2016/02/20/testing-multiple-python-versions-on-circleci.html

I hope this works for others as well!

2 Likes
12

Using pyenv to switch Python versions only worked for me after I did

machine:
  pre:
    - brew install pyenv
    - echo 'eval "$(pyenv init -)"' >> ~/.bash_profile
    - source ~/.bash_profile
13

If you’re using CCI v2.0 this works great for me, Docker ftw…

version: 2

jobs:
  build:
    working_directory: ~/your_thing
    docker:
      - image: themattrix/tox
    steps:
      - checkout
      - run: tox

More information…

3 Likes