Matt RaibleMatt Raible is a Web Architecture Consultant specializing in open source frameworks.


Over 10 years ago, I wrote my first blog post. Since then, I've authored books, had kids, traveled the world, found Trish and blogged about it all.

Web Application Testing with Selenium by Jason Huggins

Selenium This evening, I attended Agile Denver's monthly meeting to listen to Jason Huggins talk about Selenium. The meeting started off with a panel on UI testing that I participated in. The most interesting part of this panel (for me) was meeting the other panelists and learning about their expertise. Folks from Red Pine Studios in Boulder video taped both the panel and presentation. Hopefully it will be published online in the near future.

Below are my notes from Jason's talk. Please keep in mind that most of these are his words, not mine.

Jason is the Executive Software Chef at Sauce Labs. He often experiments with new recipes and is one of the creators of Selenium. He worked at Google and helped them build and use a Selenium Farm to test Gmail and Google Docs. Selenium was inspired by ThoughtWorks Expense Report system and its "Add Row" button. The button caused so many issues, they needed a way to write a test that could be run in multiple browsers.

The first thing they tried was jWebUnit (a wrapper around HtmlUnit). Since HtmlUnit simulates the browser, it wasn't "real world" enough. The 2nd attempt was DriftWood. It was a Mozilla extension that drove a real browser so it could handle JavaScript UI features. The downside it was it didn't work for IE or Safari. It also used XML Syntax for tests. The 3rd attempt was JsUnit. It worked in all browsers, but its emphasis was on a single page unit test; it had no page-to-page workflow support. Also, you couldn't see what it was doing while it was running. The 4th attempt was FIT (Framework for Integration Testing). It allowed more readable tests, but the API wasn't that intuitive and there was too much magic behind the curtain. So basically, they had to fork FIT.

The first attempt was called "Selenese" and consisted of a 3-column table where each row had an Action, Target and Value. In the beginning, Selenium Core was a TestRunner that ran in any browser. It was written in plain ol' JavaScript and HTML. The next thing that came about was the Selenium IDE for Firefox. It maintains the echo of Selenium Core and FIT.

Selenium Remote Control (RC) was the next product produced by the project. Selenium RC allows you to write your tests in any language. A Selenium server interprets the requests and turns them into browser manipulation events. Finally, Selenium Grid was developed to leverage Selenium's HTTP architecture to allow parallel execution across servers.

Cloud computing is a wonderful use case for functional testing. Selenium Hub is a gateway into the Selenium Grid that routes the test request to particular browsers and platforms. Sauce Labs has a version of Selenium Grid that runs in the cloud.

Selenium Issues
Selenium is slow. Functional tests will always be slower than unit tests. Until the browsers can launch faster, there's always going to be speed issues. Parallel-ization can solve some of these and is something you should think about right away.

The JavaScript sandbox, Flash, Java Applets, Silverlight and Canvas all present problems in Selenium. Silverlight was shipped without any testing APIs. There are several libraries that provide a bridge for testing Flash. The Selenium project has though about including FlexMonkey, but its GPL license prevents it.

Practical Advice
Everyone seems to build a framework on top of Selenium. If you do this, make sure and write your DSL in terms of intent and then map it to Selenium actions.

Look for abstractions so you're not writing your Selenium tests with its API. It's too much like Assembler.

K.I.S.S. - don't write large tests, just do small ones. Often, when functional tests fail, they tell you something failed, but they don't tell you what failed. The shortest possible functional tests help reduce the scope of where a problem can be. Other benefits of short tests are they're easier to read and easier to write.

Selenium 2.0
The big thing in Selenium 2.0 is a merger with WebDriver. The nice thing about WebDriver is it gets rid of Selenium RC and allows you to drive the browser with a low-level API. For example, you use C++ to drive IE. Basically, every language will talk to the C driver. Except for Firefox, the connection and control is done through telnet. Selenium 2 should fix all the problems with Selenium 1, but also allow you to still use Selenium RC if you want to do grid-style testing.

Selenium 2's API is about finding elements and interacting with those elements. Also, it's entirely backwards compatible, so you can use the old API.

At this point, my laptop's battery died and I was unable to take any more notes. However, I was able to see some pretty slick demos, particularly Jason's company's Sauce onDemand cloud testing services. All you need to do to run your tests in the cloud is change how you initialize Selenium. A kick-ass feature this service provides is video playback (a.k.a. Castro). I'm currently using Selenium's screenshot functionality, but it doesn't hold a candle to the ability to watch a video playback of your tests. Jason also showed us a demo of using Castro and Selenium 2 to create a screencast on-the-fly. Very cool stuff.

My only question after seeing this talk is what's the difference between BrowserMob and Sauce Labs? Both companies were founded by Selenium committers and seem to offer competing projects. My gut feel is that BrowserMob is best for performance/load testing and Sauce Labs is best for running your tests in the cloud.

Posted in Java at Feb 15 2010, 09:44:44 PM MST 5 Comments

[Trackback] This post was mentioned on Twitter by mraible: My notes from @hugs' Web Application Testing with #Selenium talk at #agiledenver:

Posted by uberVU - social comments on February 16, 2010 at 09:34 AM MST #

Having worked with both of these products they offer 2 different things from Selenium.

BrowserMob is concerned with monitoring your site and doing load and performance testings.

Sauce labs is all about running your functional tests as fast as you can by parallelizing your scripts.

Posted by David Burns on February 16, 2010 at 09:34 AM MST #

great post; thanks for summarizing. responding to the question of the difference between sauce labs and browser mob, your assessment is essentially correct. the two companies enjoy a cordial market and personal relationship. browser mob focusses on load testing (high volume of standard use cases) and uptime monitoring. sauce labs focusses on functional testing (think broad use cases and corner cases). put differently, the prototypical browser mob user is operations focussed while the sauce labs user is either development-focussed, qa-focussed, or some combination of both.

john dunham
ceo, cofounder, sauce labs inc

Posted by john dunham on February 16, 2010 at 04:19 PM MST #

Great post Matt!

We are starting to use Selenium to test our tool that was developed using GWT/GXT. I remember that in one of your posts you told something about creating fixed ids for all GXT widgets. Do you have any other experiences and best practices regarding texting GXT with Selenium?



Posted by Otavio Silva on February 21, 2010 at 01:59 PM MST #

Note that JWebUnit now wraps HtmlUnit but this was not the case at the time they tried it as you report.

Posted by Marc Guillemot on February 21, 2010 at 02:00 PM MST #

Post a Comment:
  • HTML Syntax: Allowed