Is it possible to run selenium tests with Safari under CircleCI? I have a selenium framework with python, I don’t have a mac but I need tests to be run in Safari besides Chrome. I already got a workflow that runs tests in chrome.
I’d need to do this without using emulators like suacelabs or browserstalk.
Thanks
Hello Ashea,
In the same way you can run tests in Linux containers, you can also make use of our MacOS fleet. These are true Mac machines, with safari installed.
Plan details: https://circleci.com/pricing/#build-os-x
How to write your config for MacOS: https://circleci.com/docs/2.0/executor-types/#using-macos
Selenium example using Linux: https://circleci.com/docs/2.0/browser-testing/#selenium
This should contain everything you need to get going! Please let us know if you have any further questions we can assist with.
Hi @KyleTryon ,
We finally switched to free trial macos plan. I’m trying now to get selenium tests running with Safari but it seems Safari browser in container requires allowing Automation Remote or safari driver enabling… tried different commands but I still get error message:
“WebDriverException: Message: Service /usr/bin/safaridriver unexpectedly exited. Status code was: 1”
I included following command on config already:
sudo /usr/bin/safaridriver --enable
do you know what could be missing here? Maybe I need admin access?
Thanks!
Are you trying to test an iOS app, or a MacOS app? If I recall correctly, Automation Remote is a permission needed for MacOS apps UI testing and is something we can’t support, but I’d love to be remembering wrong.
MacOS app, it’s a website that I want to test with selenium using Safari browser. What is not supported?
UI testing for a Mac app, so not related. Can you share the full log from Selenium? That may help us determine why it’s exiting.
Sure, here it is:
#!/bin/bash -eo pipefail
cd tests && python -m pytest --alluredir build/reports
^D^D============================= test session starts ==============================
platform darwin -- Python 2.7.15, pytest-4.0.2, py-1.8.0, pluggy-0.9.0
rootdir: /Users/distiller/project/tests, inifile:
plugins: metadata-1.8.0, html-1.20.0, allure-adaptor-1.7.10
collected 37 items
test_clients_01_validate_prism_from_list_and_detail_page.py E
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "/usr/local/lib/python2.7/site-packages/_pytest/main.py", line 185, in wrap_session
INTERNALERROR> session.exitstatus = doit(config, session) or 0
INTERNALERROR> File "/usr/local/lib/python2.7/site-packages/_pytest/main.py", line 225, in _main
INTERNALERROR> config.hook.pytest_runtestloop(session=session)
INTERNALERROR> File "/usr/local/lib/python2.7/site-packages/pluggy/hooks.py", line 289, in __call__
INTERNALERROR> return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR> File "/usr/local/lib/python2.7/site-packages/pluggy/manager.py", line 68, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR> File "/usr/local/lib/python2.7/site-packages/pluggy/manager.py", line 62, in <lambda>
INTERNALERROR> firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR> File "/usr/local/lib/python2.7/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR> return outcome.get_result()
INTERNALERROR> File "/usr/local/lib/python2.7/site-packages/pluggy/callers.py", line 81, in get_result
INTERNALERROR> _reraise(*ex) # noqa
INTERNALERROR> File "/usr/local/lib/python2.7/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR> res = hook_impl.function(*args)
INTERNALERROR> File "/usr/local/lib/python2.7/site-packages/_pytest/main.py", line 246, in pytest_runtestloop
INTERNALERROR> item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR> File "/usr/local/lib/python2.7/site-packages/pluggy/hooks.py", line 289, in __call__
INTERNALERROR> return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR> File "/usr/local/lib/python2.7/site-packages/pluggy/manager.py", line 68, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR> File "/usr/local/lib/python2.7/site-packages/pluggy/manager.py", line 62, in <lambda>
INTERNALERROR> firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR> File "/usr/local/lib/python2.7/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR> return outcome.get_result()
INTERNALERROR> File "/usr/local/lib/python2.7/site-packages/pluggy/callers.py", line 81, in get_result
INTERNALERROR> _reraise(*ex) # noqa
INTERNALERROR> File "/usr/local/lib/python2.7/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR> res = hook_impl.function(*args)
INTERNALERROR> File "/usr/local/lib/python2.7/site-packages/_pytest/runner.py", line 76, in pytest_runtest_protocol
INTERNALERROR> runtestprotocol(item, nextitem=nextitem)
INTERNALERROR> File "/usr/local/lib/python2.7/site-packages/_pytest/runner.py", line 85, in runtestprotocol
INTERNALERROR> rep = call_and_report(item, "setup", log)
INTERNALERROR> File "/usr/local/lib/python2.7/site-packages/_pytest/runner.py", line 177, in call_and_report
INTERNALERROR> hook.pytest_exception_interact(node=item, call=call, report=report)
INTERNALERROR> File "/usr/local/lib/python2.7/site-packages/pluggy/hooks.py", line 289, in __call__
INTERNALERROR> return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR> File "/usr/local/lib/python2.7/site-packages/pluggy/manager.py", line 68, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR> File "/usr/local/lib/python2.7/site-packages/pluggy/manager.py", line 62, in <lambda>
INTERNALERROR> firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR> File "/usr/local/lib/python2.7/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR> return outcome.get_result()
INTERNALERROR> File "/usr/local/lib/python2.7/site-packages/pluggy/callers.py", line 81, in get_result
INTERNALERROR> _reraise(*ex) # noqa
INTERNALERROR> File "/usr/local/lib/python2.7/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR> res = hook_impl.function(*args)
INTERNALERROR> File "/Users/distiller/project/tests/conftest.py", line 12, in pytest_exception_interact
INTERNALERROR> contents=service_page.get_screenshot(),
INTERNALERROR> File "/Users/distiller/project/pages/advisors_stratifi_pages/service_page.py", line 48, in get_screenshot
INTERNALERROR> screenshot = self._driver.get_screenshot_as_png()
INTERNALERROR> File "/usr/local/lib/python2.7/site-packages/webium/base_page.py", line 52, in _driver
INTERNALERROR> return get_driver()
INTERNALERROR> File "/usr/local/lib/python2.7/site-packages/webium/driver.py", line 10, in get_driver
INTERNALERROR> _driver_instance = webium.settings.driver_class()
INTERNALERROR> File "/usr/local/lib/python2.7/site-packages/selenium/webdriver/safari/webdriver.py", line 57, in __init__
INTERNALERROR> self.service.start()
INTERNALERROR> File "/usr/local/lib/python2.7/site-packages/selenium/webdriver/common/service.py", line 98, in start
INTERNALERROR> self.assert_process_still_running()
INTERNALERROR> File "/usr/local/lib/python2.7/site-packages/selenium/webdriver/common/service.py", line 111, in assert_process_still_running
INTERNALERROR> % (self.path, return_code)
INTERNALERROR> WebDriverException: Message: Service /usr/bin/safaridriver unexpectedly exited. Status code was: 1
=========================== 1 error in 6.28 seconds ============================
Exited with code 3
Also here are commands I use in config.yml file in case you need it:
#!/bin/bash -eo pipefail
xcodebuild -version
pip install -U selenium
pip install --upgrade pip
pip install -U webium
pip install pytest==4.0.2
pip install -U pytest-allure-adaptor
pip install pytest-html
pip install pyperclip==1.5.27
pip install seleniumwrapper
pip install pycrypto
pip install requests
brew install pigz
python -c "import selenium; print(selenium.__version__)"
brew install qatools/formulas/allure-commandline
sudo /usr/bin/safaridriver --enable
Huge Thanks!
I’m not seeing anything that stands out. 
One thing that may help is to try troubleshooting with VNC and SSH https://circleci.com/docs/2.0/browser-testing/#interacting-with-the-browser-over-vnc
That will allow you to view the tests and may give insight into what’s happening.
Thanks @drazisil will try that, in the meantime if you come up with any other idea that could fix this will be very much appreciated. Thanks
Hi @drazisil, I see that https://circleci.com/docs/2.0/browser-testing/#interacting-with-the-browser-over-vnc steps are all for a case accessing linux containers, but in this case I need to access to a Mac.
Do you know if there are proper instruccions for that? For example which is the username in Mac?
ssh -p PORT ubuntu@IP_ADDRESS -L 5902:localhost:5901 what should I use instead of Ubuntu?
Thanks in advance!
The instructions will be mostly the same. The SSH login info is provided when you choose SSH build, under the enable SSH step.
Hi, I managed to ssh connect to the mac container build.
Thing now is, I need VCN server running on the host, apparently from what I read it is already included on MacOS builts since 10.4 version so it should be there … BUT I’ve read you need to enable it from preferences tab… so do I need somehow to do that in the remote mac build or is just automatically enabled together with the ssh connection?
Also following step here is:
ubuntu@box159:~$ vncserver -geometry 1280x1024 -depth 24
Again… how this command could work for mac? tried replacing ubuntu with my username but no luck, maybe I still need vnc server enabling…
Please let me know how to proceed, I’ve got only couple of days with trial macos plan to figure this out and it won’t be worth switching to macos plan if we are not sure about this working.
Thanks!
I don’t work for CircleCI, but it might be worth shooting them a quick email to see if you can get an extension on your trial. Worth a try? 
Hi Ash,
That step should be run once you are logged into an SSH build, so the ubuntu@box159:~$ is a prompt, and not part of the command. The command you would run on the box is vncserver -geometry 1280x1024 -depth 24
Ok makes sense, thanks. I tried vncserver -geometry 1280x1024 -depth 24 command being ssh connected to the build but it is not recognizing vncserver command… so back to my first question, how do I enable vncserver on remote host?
Hi @dbrgn, no… I’m trying now to view remotely what’s going on in macos container connecting by ssh but still couldn’t even get to Safari to see why it is giving that error.
were you able to get a desktop connection to a macos container? I am trying to login to see if I can debug an issue I am having
Hi folks. I might have led you astray here. 
It’s possible that VNC isn’t an option on OSX and it was wishful thinking on my part. I’m going on vacation so I can’t look into this for a while, but can I ask you to open a ticket so our support team can look into this please?
We have a support center article here https://support.circleci.com/hc/en-us/articles/360020345334-How-do-I-connect-mac-container-via-VNC-
Apparently it’s not nearly as easy to do this on OSX as it is on Linux.