Web tests investigation

[cneves]

The web framework has multiple parts that need to be tested separately;

• Rendered Templates: requires something like Selenium
• Javascript: these get exercised when testing Rendered Templates, but might be worth to test them specifically using something like QUnit
• Django Views: per django project, we're likely to continue django's unittest framework
• Python Gateway: this is likely to stay a part of the global omero unit testing, but needs a lot of TLC

First step is getting acquainted with the options we have wrt tools, particularly for the Selenium use case.

[cneves]

On the Rendered Templates testing, there are actually two separate types of testing;

• Rendering consistency across browsers - This is something we haven't automated and I don't thing this is an important step at this stage
• Functional test of pages across browsers - Where Selenium shines, and the one I'm investigating currently. There are both tools and services that can be used.

[cneves]

For rendered template testing, i.e. user interaction tests I have found 3 good options:

• ​http://Watir.com - there's a good community effort behind it, and seems to be comprehensive. Controls browsers using webdriver. The big caveat is that tests are written in ruby, which is not by itself a problem but most people on the project are more familiar with python.
• ​http://getwindmill.com - Windmill looks awesome, and given the attention they draw to some features like drag & drop testing for which selenium has been a pain to use in the past, I would certainly like to take this one for a spin. It has an IDE for preparing tests and the end test files are simple python scripts.
• ​http://docs.seleniumhq.org - We have a rough framework for using selenium already. When we used it selenium worked by opening a browser window and injecting some javascript that would then control the page, which presented some issues. Now fully supports webdriver, both locally and remote, has full support in python and can now be run without first starting any server when testing locally. This is by far the simplest approach for testing and it has an extra feature to it: ​https://saucelabs.com/ where you can run your tests on a number of OS/Browser pairs (these are just selenium tests, may even work unchanged), get the testing results and video of complete testing session.

While I am torn between windmill and selenium, the fact we can run tests on local machines while developing, run them across a set of testing servers we manage ourselves or use a service like saucelabs (there is at least on more such service than also takes selenium tests, but I can't find it again for some reason) make selenium (2+) the absolute winner in my mind.

For javascript unit testing there are also good options, but as I have no previous experience in any of them I'd propose we try to use ​http://qunitjs.com/ as this was written by the author of jquery for jquery, although it is now an independent project for javascript unit testing.

Finally, there is ​http://phantomjs.org/ which is a complete scriptable webkit framework. This is pretty impressive as it can be used for headless testing for the complete stack and can be used with other testing frameworks but since it is webkit specific and we need to test across browsers it might not be the best place to put our effort on. It would be amazing coupled with our continuous integration framework though!

[wmoore]

I'll try and get our existing selenium tests up and running, beginning at #3299.

Following docs at ​http://docs.seleniumhq.org/docs/03_webdriver.jsp, install selenium for Python:

$ pip install selenium
Downloading/unpacking selenium
  Downloading selenium-2.33.0.tar.gz (2.5Mb): 2.5Mb downloaded
  Running setup.py egg_info for package selenium
Installing collected packages: selenium
  Running setup.py install for selenium
Successfully installed selenium
Cleaning up...

Then I removed the old dist/lib/python/selenium.py so that import selenium imports the new selenium above

However, with webclient running on localhost 4080, this failed (actually got the same error using selenium

$ omero web seleniumtest webadmin localhost http://localhost:4080 firefox

======================================================================
ERROR: testCreateExperimenter (__main__.AdminTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/will/Desktop/OMERO/dist/lib/python/omeroweb/webadmin/tests/seleniumtests.py", line 112, in setUp
    super(AdminTests, self).setUp()
  File "/Users/will/Desktop/OMERO/dist/lib/python/omeroweb/webgateway/tests/seleniumbase.py", line 142, in setUp
    self.selenium = self.SERVER.getSelenium()
  File "/Users/will/Desktop/OMERO/dist/lib/python/omeroweb/webgateway/tests/seleniumbase.py", line 73, in getSelenium
    self.selenium.start()
  File "/Users/will/apps/OMERO.libs/lib/python2.6/site-packages/selenium/selenium.py", line 198, in start
    result = self.get_string("getNewBrowserSession", start_args)
  File "/Users/will/apps/OMERO.libs/lib/python2.6/site-packages/selenium/selenium.py", line 232, in get_string
    result = self.do_command(verb, args)
  File "/Users/will/apps/OMERO.libs/lib/python2.6/site-packages/selenium/selenium.py", line 221, in do_command
    conn.request("POST", "/selenium-server/driver/", body, headers)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/httplib.py", line 874, in request
    self._send_request(method, url, body, headers)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/httplib.py", line 911, in _send_request
    self.endheaders()
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/httplib.py", line 868, in endheaders
    self._send_output()
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/httplib.py", line 740, in _send_output
    self.send(msg)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/httplib.py", line 699, in send
    self.connect()
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/httplib.py", line 683, in connect
    self.timeout)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/socket.py", line 512, in create_connection
    raise error, msg
error: [Errno 61] Connection refused

Then made a couple of changes to dist/lib/python/omeroweb/webgateway/tests/seleniumbase.py

+ from selenium import driver

-         self.selenium.start()
+        self.selenium.start(driver=webdriver.Firefox())

This actually got Firefox browsers opening for each test, but failed with

======================================================================
ERROR: testCreateExperimenter (__main__.AdminTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/will/Desktop/OMERO/dist/lib/python/omeroweb/webadmin/tests/seleniumtests.py", line 112, in setUp
    super(AdminTests, self).setUp()
  File "/Users/will/Desktop/OMERO/dist/lib/python/omeroweb/webgateway/tests/seleniumbase.py", line 142, in setUp
    self.selenium = self.SERVER.getSelenium()
  File "/Users/will/Desktop/OMERO/dist/lib/python/omeroweb/webgateway/tests/seleniumbase.py", line 72, in getSelenium
    self.selenium.start(driver=webdriver.Firefox())
  File "/Users/will/apps/OMERO.libs/lib/python2.6/site-packages/selenium/selenium.py", line 196, in start
    id = driver.desired_capabilities['webdriver.remote.sessionid']
KeyError: u'webdriver.remote.sessionid'

Not sure what to try now ;(

[wmoore]

However, based on the example at ​http://docs.seleniumhq.org/docs/03_webdriver.jsp#webdriver-and-the-selenium-server, this works fine:

from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait # available since 2.4.0
from selenium.webdriver.support import expected_conditions as EC # available since 2.26.0

# Create a new instance of the Firefox driver
driver = webdriver.Firefox()

# go to the webclient login page
driver.get("http://localhost:4080/webclient/")

# find fields and enter login details
inputName = driver.find_element_by_name("username")
inputName.send_keys("will")
driver.find_element_by_name("password").send_keys("ome")

print driver.title

# submit the form
inputName.submit()

try:
    # we have to wait for the page to refresh, the last thing that seems to be updated is the title
    WebDriverWait(driver, 10).until(EC.title_contains("Webclient"))

    # You should see "cheese! - Google Search"
    print driver.title

finally:
    driver.quit()

[cneves]

On to unit / funtional python testing.

We've been using the python included unittest module, which is a fine choice and has served us quite well, but since it can't hurt to know what else is out there, I've looked into alternatives and while there are many other frameworks, two of them stand out, nose and pytest.

• ​https://nose.readthedocs.org/en/latest/
• ​http://pytest.org/latest/

The two tools are very similar and both aim at removing boilerplate code and automating test detection. Both have plugin support and also have django integration projects that are active.

First, one needs to consider if there's really a need to change the testing framework from unittest, and I must say that the fact we don't have to maintain test suites and the much simpler test code are by themselves more than enough reason. I mean, when writing tests becomes simpler and easier there's a much better change tests will be written, right?

But there are other good points to these libraries, one of them is the setup and destroy hooks that are much more broad than in unittests. This is particularly important when testing the django app side, where each test module may need specific setup steps that will be used throughout all the specific modules, and with unittests one ends up having to hack that behavior which is fine and it works, but is a little added "magic" that may not be obvious to others reading or extending the tests.
Tests no longer need to be organized in classes, although that is supported. Both libraries support running unittest and doctest tests.

On the more exotic side there is one particular plugin class that exists in both libraries, if with a little different implementation, and that's multiprocess test runs. Pytest takes that a step forward and allows for easy remote run of tests which is an amazing tool for continuous integration and test running in multiple platforms. Specific to pytest as far as I could understand (although I'm sure it would be possible to add to nose if not already there) is the loop on failing feature, where when a set of tests fail they will be rerun until they pass, triggered by changes to the test files.

• ​http://nose.readthedocs.org/en/latest/plugins/multiprocess.html
• ​http://pytest.org/latest/xdist.html

I grew a personal bias towards pytest because it has more setup / destroy hooks (which is often described as a fault for some reason) and because the xdist plugin just plain rocks.

There's also unittest2 which is standard with python2.7+ that already has many of the features introduced by nose and pytest, but we need to support older versions of python and thus I don't think it is practical to even consider it.

Below is a link to a comparison made between nose and pytest:

• ​http://fedoraproject.org/wiki/User:Tflink/AutoQA_nose_pytest_comparison

[jburel]

As indicated on #11088 and during Tuesday meeting, I am investigating the RobotFramework to run selenium tests.
Firefox needs to be installed. Do no install Firefox version 22 or 23, issue with event and Path.

A connection test will look like

Valid Login
    Open Browser To Login Page
    Input Username    root
    Input Password    omero
    Submit Credentials
    Welcome Page Should Be Open
    [Teardown]  Close Browser

[wmoore]

To conclude the investigation, we have started using Robot Framework with Selenium 2.