Integrating Selenium with Maven 2
I spent some time this past week integrating Selenium with Maven 2. This post is designed to show you how to do this in your Maven 2 projects.
First of all, there were two types of testing scenarios I wanted to make possible. The first was to allow HTML-based tests that web designers could create and run with Selenium IDE. As far as I know, Selenium IDE is capable of recording and exporting Java-based tests (powered by TestNG or JUnit), but I don't believe it's capable of playing them back. So for Java Developers, I wanted to allow them to write their tests in Java.
To get Maven to run HTML-based tests, the easiest way seems to be using the <selenese> Ant task. I tried Mavenium as well, but it 1) doesn't use the latest version of Selenium RC and 2) reports success when tests fail. Below is a Maven profile that I'm using in an AppFuse-based project to run HTML tests.
<profiles>
<profile>
<id>integration-test</id>
<activation>
<property>
<name>!maven.test.skip</name>
</property>
</activation>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.cargo</groupId>
<artifactId>cargo-maven2-plugin</artifactId>
<version>0.3-SNAPSHOT</version>
<configuration>
<wait>${cargo.wait}</wait>
<container>
<containerId>${cargo.container}</containerId>
<!--home>${cargo.container.home}</home-->
<zipUrlInstaller>
<url>${cargo.container.url}</url>
<installDir>${installDir}</installDir>
</zipUrlInstaller>
</container>
<configuration>
<home>${project.build.directory}/${cargo.container}/container</home>
<properties>
<cargo.hostname>${cargo.host}</cargo.hostname>
<cargo.servlet.port>${cargo.port}</cargo.servlet.port>
</properties>
</configuration>
</configuration>
<executions>
<execution>
<id>start-container</id>
<phase>pre-integration-test</phase>
<goals>
<goal>start</goal>
</goals>
</execution>
<execution>
<id>stop-container</id>
<phase>post-integration-test</phase>
<goals>
<goal>stop</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-antrun-plugin</artifactId>
<executions>
<execution>
<id>launch-selenium</id>
<phase>integration-test</phase>
<configuration>
<tasks>
<taskdef resource="selenium-ant.properties">
<classpath refid="maven.plugin.classpath"/>
</taskdef>
<selenese suite="src/test/resources/selenium/TestSuite.html"
browser="*firefox" timeoutInSeconds="180" port="5555"
results="${project.build.directory}/selenium-firefox-results.html"
startURL="http://${cargo.host}:${cargo.port}/${project.build.finalName}/"/>
</tasks>
</configuration>
<goals>
<goal>run</goal>
</goals>
</execution>
</executions>
<dependencies>
<dependency>
<groupId>ant</groupId>
<artifactId>ant-nodeps</artifactId>
<version>1.6.5</version>
</dependency>
<dependency>
<groupId>org.openqa.selenium.server</groupId>
<artifactId>selenium-server</artifactId>
<version>0.9.1-SNAPSHOT</version>
</dependency>
</dependencies>
</plugin>
</plugins>
</build>
</profile>
<profile>
<id>selenium-ie</id>
<activation>
<os>
<family>windows</family>
</os>
</activation>
<build>
<plugins>
<plugin>
<artifactId>maven-antrun-plugin</artifactId>
<executions>
<execution>
<id>launch-selenium</id>
<phase>integration-test</phase>
<configuration>
<tasks>
<taskdef resource="selenium-ant.properties">
<classpath refid="maven.plugin.classpath"/>
</taskdef>
<selenese suite="src/test/resources/selenium/TestSuite.html"
browser="*firefox" timeoutInSeconds="180" port="5555"
results="${project.build.directory}/selenium-firefox-results.html"
startURL="http://${cargo.host}:${cargo.port}/${project.build.finalName}/"/>
<selenese suite="src/test/resources/selenium/TestSuite.html"
browser="*iexplore" timeoutInSeconds="180" port="5555"
results="${project.build.directory}/selenium-ie-results.html"
startURL="http://${cargo.host}:${cargo.port}/${project.build.finalName}/"/>
</tasks>
</configuration>
<goals>
<goal>run</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
The above setup will allow you to run Selenium tests in Firefox, and in IE as well when you're on Windows. I tried to get Safari to work on the Mac, but it just opens Safari and hangs.
HTML tests are great for non-programmers, but what about developers that prefer Java and want test reports to be included in the surefire-plugin's reports? That's easy enough. First of all, put your tests in a particular package so they can be excluded from the normal testing cycle. I used a webapp.selenium package. I configured the surefire-plugin to exclude these tests:
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<excludes>
<exclude>**/selenium/*Test.java</exclude>
</excludes>
</configuration>
</plugin>
Then I added the newly released selenium-maven-plugin to my "integration-test" profile and configured surefire to run the Selenium Java tests.
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>selenium-maven-plugin</artifactId>
<version>1.0-beta-1</version>
<executions>
<execution>
<id>start-selenium</id>
<phase>pre-integration-test</phase>
<goals>
<goal>start-server</goal>
</goals>
<configuration>
<background>true</background>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<executions>
<execution>
<id>surefire-it</id>
<phase>integration-test</phase>
<goals>
<goal>test</goal>
</goals>
<configuration>
<excludes>
<exclude>none</exclude>
</excludes>
<includes>
<include>**/selenium/*Test.java</include>
</includes>
</configuration>
</execution>
</executions>
</plugin>
Yeah, Maven can be quite verbose when configuring profiles. I contacted the Maven list to see if it's possible to simplify all this XML, but so far haven't found a solution.
If you'd like to see a pom.xml with the Selenium bits and a profile that runs both HTML and Java-based tests, click here (JavaScript needs to be enabled for this to work).
NOTE: I used 0.9.1-SNAPSHOT of Selenium Server because it solves issues with the latest version of Firefox.
This brings up a related question I asked on the AppFuse mailing list a couple of days ago:
Do you use the Canoo WebTests? If not, how do you do UI testing? If so, would you prefer Selenium?
If you've tried AppFuse 2.x and have an opinion, please add a comment. Personally, I like Selenium, but I like how Canoo WebTest can be somewhat friendly to designers and allow i18n testing with Ant's property file support. With Selenium, you have to use parse/replace or Java tests to do i18n testing. Then again, if you need to test a lot of Ajax functionality, it's likely that Selenium will work much better for you.
Posted by Howard Lewis Ship on March 09, 2007 at 03:13 PM MST #
Posted by sebnoumea on March 09, 2007 at 05:18 PM MST #
Posted by Charles Crouch on March 09, 2007 at 05:31 PM MST #
Posted by Paul Hammant on March 11, 2007 at 08:26 AM MDT #
Posted by Carlos Sanchez on March 12, 2007 at 07:44 PM MDT #
Posted by ros on June 21, 2007 at 04:29 PM MDT #
Posted by Matt Raible on June 21, 2007 at 04:33 PM MDT #
Posted by Confluence: WEB2.0 on July 16, 2007 at 09:14 AM MDT #
Thanks for the detailed example.
In order to get a very similar pom to run with the selenese ant task under windows I had to change the used ant taskdef-classpathref to maven.test.classpath (in your example it's maven.plugin.classpath). maven.plugin.classpath includes under windows only the core Ant and Maven jars. The Selenium jars (with the SeleneseAntTask) are missing -- don't know why, with MacOSX maven.plugin.classpath ran fine.
Posted by Jonas Fleer on August 09, 2007 at 09:36 AM MDT #
I would like to contribute a very important addendum to this tutorial. If you provide custom excludes for the maven-surefire-plugin, as you have done to exclude the selenium tests from the unit test run, it is imperative that you restore the inner class filter, or else the runner will fail attempting to scan inner classes for test methods.
<plugin> <artifactId>maven-surefire-plugin</artifactId> <configuration> <excludes> <exclude>**/selenium/*Test.java</exclude> <exclude>**/*$*</exclude> </excludes> </configuration> </plugin>Posted by Dan Allen on March 14, 2008 at 01:34 PM MDT #
At kicktipp we ran into a few problems while running our intergation tests: we don't have a x-Server on our Continous integration (CI) server and wanted to run selenes HTML Tests as some more complex Selenium JUnit Tests using the selenium RC.
If running on linux you won't always have a running X-Server for starting firefox, especially when running on a CI. Therefore you need a Xvfb. Add this execution to your selenium plugin:
<execution> <id>xvfb</id> <phase>pre-integration-test</phase> <goals> <goal>xvfb</goal> </goals> </execution>Additionally you don't need an ant task if you want to run your selenese Tests, you can use selenium:selenese goal:
<execution> <id>selenese</id> <phase>integration-test</phase> <goals> <goal>selenese</goal> </goals> <configuration> <port>4445</port> <browser>*firefox</browser> <suite>src/test/resources/tipprunde/tippspiel.html</suite> <startURL>http://localhost:8080/</startURL> </configuration> </execution>You need a diffrent port here as the selenium-start-server goal is still executing on port 4444 and selenese does start its own server.
But there is another problem to fix: the selenese goal does not respect the running xvfb. Other than the server-start goal it does not read the display properties from target/selenium/display.properties. As there is no way to pass the display to this goal, you should set an environment variable
in your start-up script of your favourite CI-serverIt worked for us at least.
Posted by Janning Vygen on October 08, 2008 at 03:53 AM MDT #
Posted by iLinkShare (Web 2.0 linksharing) on October 31, 2008 at 11:03 AM MDT #
Posted by Aaron Stewart on December 12, 2008 at 12:40 PM MST #
Hi,
I see that your startUrl looks like this: startURL="http://${cargo.host}:${cargo.port}/${project.build.finalName}/
isn't the last part stripped off? so that it will look like startURL="http://${cargo.host}:${cargo.port}/ ? It does not produce a error.
In my case i kinda need this but it is stripped off. Anyone have more information about this?
Posted by kukudas on February 19, 2009 at 09:17 AM MST #
If you want to deploy your app at ROOT (which I didn't), you can use the following configuration.
<plugin> <groupId>org.codehaus.cargo</groupId> <artifactId>cargo-maven2-plugin</artifactId> <version>1.0-beta-2</version> <configuration> <wait>${cargo.wait}</wait> <container> <containerId>${cargo.container}</containerId> <!--home>${cargo.container.home}</home--> <zipUrlInstaller> <url>${cargo.container.url}</url> <installDir>${installDir}</installDir> </zipUrlInstaller> </container> <configuration> <home>${project.build.directory}/${cargo.container}/container</home> <properties> <cargo.hostname>${cargo.host}</cargo.hostname> <cargo.servlet.port>${cargo.port}</cargo.servlet.port> </properties> <deployables> <deployable> <properties> <context>ROOT</context> </properties> </deployable> </deployables> </configuration> </configuration> <executions> <execution> <id>start-container</id> <phase>pre-integration-test</phase> <goals> <goal>start</goal> </goals> </execution> <execution> <id>stop-container</id> <phase>post-integration-test</phase> <goals> <goal>stop</goal> </goals> </execution> </executions> </plugin>Posted by Matt Raible on February 19, 2009 at 11:40 AM MST #
Posted by kukudas on February 20, 2009 at 02:01 AM MST #
Precisely what I was looking for. Why are there so many posts (including the core documentation for the maven-selenium plugin) that talk about how to *disable* unit tests so you can do integration tests with Selenium? Why is that useful at all?
It makes so much more sense to properly filter your tests into those you want to run per lifecycle stage (i.e. test vs integration-test) and use profiles to invoke the appropriate run.
For example, I've set up four profiles (dev,test,stage,prod) and I want dev to just run unit tests for the "test" goal plus a core set of sanity checks on the app with Selenium for the "integration-test". The 'test' and 'stage' profiles run everything together and modify some properties to incorporate more external components, including changes per environment from the QA workbench to the staging server. Finally 'prod' ONLY runs the lightweight sanity checks on Selenium for two key browser profiles (ff and ie).
With this system we can do full test runs in about 5 min (expect to take 20+ once more tests are checked in) but I can do a complete code deployment plus sanity check for critical use cases in < 5 min. If it fails, I just invoke my rollback scripts and try again once the issues are ironed out.
BTW, yes we made the decision to make the app unavailable to the world (i.e. maintenance page) during production rollouts and force a 5 min automated deployment window. Trying to rollback db changes while live users are committing transactions is the quintessential definition of 'hell'.
Posted by Adam McClure on August 18, 2009 at 01:42 PM MDT #