1

Selenium, Python, and Sauce

With a title like that most people are probably thinking I have some sort of strange fetish — but alas, my sickness is far geekier. I’m talking of course about User Acceptance Testing (UAT). Selenium, for those of you who are unaware, is an automated testing system that will run assertions against a web site. In our case, we use Python to invoke these browser commands but there are many other languages you can write them in.

Since there are many posts about this subject (sources listed at the end) I am not going to chronicle all the intricacies of the entire setup, rather some of the tricks I used and my experiences with it.

At Buyer’s Best Friend we are a Python shop. At first when I brought Selenium to the company I wrote them in HTML to demonstrate their effectiveness as tool to replace the lack of QA and speedup our release process. Once their power was demonstrated naturally I chose Python to keep one common language across our codebases.

This added a interesting challenge however. Since we use Python 2.5 we cannot run them concurrently. To overcome this we run our tests using 2.6 and added the nosetest framework. For a more detailed explanation, read Running Your Selenium Tests in parallel: Python on the Sauce Labs blog.

Although Nosetests are a great tool, it adds another level of abstraction on top of my homemade test harness which I had to throwout due to the introspective nature of Nosetests and the way it wanted to run. One of the main issues I came up with was configuration. I ultimately I made my own configuration which was called in the setUp() method of each test. In my case, my base test would call it and set the config object as a member variable on the test itself. Here is my config.py:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

import ConfigParser
 
class Config():
  config = ConfigParser.ConfigParser()
  config.read("../config.cfg")
 
  if "config" not in config.sections(): 
    config = ConfigParser.RawConfigParser()
    config.add_section('config')
    config.set('config', 'host', '10.0.1.20')
    config.set('config', 'port', '4444')
    config.set('config', 'browser', 'iehta')
    config.set('config', 'target', 'my-test-server.com')
    config.add_section('sauce')
    config.set('sauce', 'enabled', 'true')
    config.set('sauce', 'browser-version', '8.')
    with open("../config.cfg", "wb") as configfile:
      config.write(configfile)
 
  # our member variables
  host = config.get('config', 'host')
  port = config.get('config', 'port')
  browser = config.get('config', 'browser')
  target = config.get('config', 'target')
 
  sauce_enabled = config.get('sauce', 'enabled')
  browser_version = config.get('sauce', 'browser-version')

As you can see from this configuration, I have two setups here. One more testing locally, or remotely against and instance or grid, or to run them on Sauce. More on that later.

On the grid

As you create more and more tests it becomes increasingly slow to run them on just one machine. Recently it was taking us 30 minutes to run our entire suite of 62 tests and counting. Enter the Selenium Grid. The grid allows you to run many machines or virtual machines and distribute your tests across a farm. Currently I run them across a laptop or two and I have been able to cut the total runtime nearly in half. Although thats good I know I can do better and deal with less configuration and hardware hassle.

Hitting the Sauce

Sauce Labs, founded by Selenium’s creator Jason Huggins, allows you to run your tests in the cloud. Not only that, they also give you an awesome dashboard, a video recording of each of your tests that run, and logging and diagnostic tools to make your tests run faster.

The configuration is brain-dead easy and only takes a matter of minutes to switch your current setup to a sauce setup. Sauce has a setup wizard to help guide you through the process but here’s the configuration I ended up going with. Below is my BaseTest class which extends unittest.TestCase. This ties in with the configuration I setup in the excerpt above.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

def setUp(self):
  self.config = Config() # from config.py (see above excerpt)
  self.verificationErrors = []
 
  if self.config.sauce_enabled:
    sauce_config = {}
    sauce_config["username"] = "john"
    sauce_config["access-key"] = "xxx"
    sauce_config["os"] = "Windows 2003"
    sauce_config["browser"] = self.config.browser
    sauce_config["browser-version"] = 
      self.config.browser_version
    sauce_config["name"] = self._testMethodName
    self.selenium = 
      selenium('ondemand.saucelabs.com', 80, 
        json.dumps(sauce_config),
        "http://" + self.config.target)
  else:
    self.selenium = 
      selenium(self.config.host, self.config.port, 
        "*" + self.config.browser,
        "http://" + self.config.target)
 
  self.selenium.start()
  self.selenium.set_timeout(90000)

As you can see from the above excerpt, you just need to instantiate your selenium object with a different configuration and you’re running Selenium in the cloud. Now I have my tests running in less than 10 minutes running 4 concurrently.

Conclusion

Selenium is a wonderful tool that I have used many times on the job, this time extensively. It has not only caught more bugs than any other human in our company, it has sped up our release process, and also greatly added to our confidence level at every push. Soon we will most likely add it to our continuous integration to get even earlier warnings. In my mind, Selenium is the key to getting repeatable successful launches week after week — with or without a dedicated QA.

In regards to running your tests, Sauce’s Selenium Grid in the cloud has been a great experience and something I am going to continue to use. I have pushed two major releases using it and I have nothing but great things to say. If you’re considering Selenium, you better consider Sauce otherwise be prepared for more configuration and wiring. Running the grid on your own sure does work and I may try it for continuous integration but Sauce is sure making it hard to go back to running my own machines!

Sources:
Selenium Official Site
Nosetests – Python Unit Test Framework
“Running Selenium Test on Sauce Labs”, Matt Raible
“Running Your Selenium Tests in parallel: Python”, Santiago Suarez Ordoñez

Categories: Computers, Software, Web