Raible Designs

Upgraded to Apache Roller 5.0, RC4

Last Sunday, Dave Johnson released Apache Roller 5.0, RC4. Since I'm an Apache Roller committer, and I've been seeing issues with comments throwing exceptions on this site, I decided to upgrade. In doing so, I discovered a number of issues. Hopefully by documenting them here, you'll be able to upgrade from Roller 4 to Roller 5 without any issues.

To upgrade, I took a snapshot of my existing site and database and copied them locally. After getting everything setup locally (importing existing database and upgrading it), I started Tomcat and began solving problems.

Database settings - JNDI vs. Properties
With Roller 4, I configured by database settings in a ROOT.xml so they'd be read using JNDI. With Roller 5, I got the following error when I tried to do this.

<openjpa-2.0.1-r422266:989424 fatal user error> org.apache.openjpa.persistence.ArgumentException:
A JDBC Driver or DataSource class name must be specified in the ConnectionDriverName property.
       at org.apache.openjpa.jdbc.schema.DataSourceFactory.newDataSource(DataSourceFactory.java:76)

I was able to get around this issue by adding the following to my roller-custom.properties:

database.configurationType=jdbc
database.jdbc.driverClass=com.mysql.jdbc.Driver
database.jdbc.connectionURL=jdbc:mysql://localhost/rollerdb
database.jdbc.username=root
database.jdbc.password=
mail.configurationType=properties
mail.hostname=localhost

After making this change, I received an error when Planet tried to startup:

ERROR 2011-03-02 09:56:08,502 DatabaseProvider:errorMessage - ERROR: unable to obtain database connection. 
Likely problem: bad connection parameters or database unavailable.
FATAL 2011-03-02 09:56:08,502 RollerContext:contextInitialized - Roller Planet startup failed during app preparation
org.apache.roller.planet.business.startup.StartupException: ERROR: unable to obtain database connection. 
Likely problem: bad connection parameters or database unavailable.

I don't remember why I enabled planet, but turning it off in roller-custom.properties seemed to solve the problem.

planet.aggregator.enabled=false

Password Encyrption
The next thing I tried to do was login. When this didn't work, I figured it must be related to password encryption. With Roller 4, I had to have "passwds.encryption.enabled=true" in roller-custom.properties. In Roller 5, I also had to add the encryption algorithm.

passwds.encryption.algorithm=SHA

GZip Compression
In November 2009, I optimized this site and used Roller's CompressionFilter and wro4j to gzip and concatenate JavaScript and CSS. With Roller 4, I used the CompressionFilter to compress *.css and *.js instead of using Wro4J's built-in gzip compression. The Roller 5 CompressionFilter seems to have issues with wro4j, so I had to disable it for *.css and *.js and use wro4j instead.

At this point, I figured I was good to go, so I zipped up my local WAR and scp'ed it to raibledesigns.com. I stopped Tomcat and attempted to upgrade my production MySQL database (version 3.23.56). Below is the error I received.

$ mysql -u raible -p raible < 400-to-500-migration.sql
Enter password:
ERROR 1064 (00000) at line 42: You have an error in your SQL syntax near 'as w set
   lastmodified = lastmodified,
   datecreated = datecreated,
   cr' at line 1

At this point, I figured my database might be slightly hosed, but since it was simply creating tables, I was probably OK. I restarted Tomcat and left the old version in place while I waited for a MySQL 5 database instance from my hosting provider, KGB Internet. Once I got the new instance, I imported my backed-up database, ran the upgrade script and everything worked just peachy.

I generally upgrade Roller by coping the new codebase over my old one. This is because I have a lot of symlinks and other files in my "ROOT" directory and like to keep those. In doing this, I found I had to do a couple things after copying everything over:

1. Delete WEB-INF/lib and recopy from RC4's WEB-INF/lib.
2. Delete WEB-INF/classes and recopy from RC4's WEB-INF/classes.

I then experienced some issues with JARs not being present for Roller's JSPWikiPlugin. I enabled this long ago, but don't use it anymore. However, to keep old posts still working, I wanted to enable it. The downloads for the plugin seem to be gone, but luckily I found a copy and put all the JARs into my WEB-INF/lib directory.

After starting Tomcat and browsing around a bit, I discovered two more issues:

1. Search doesn't seem to work. For example, there are no results for jQuery.
2. My Archives page's calendar didn't work. It showed the following:

   $calendarModel.showWeblogEntryCalendarBig($weblog, $cat)

I was able to fix issue #2 by changing #showBigWeblogCalendar() to the following.

#showWeblogEntryCalendarBig($model.weblog "nil")

The first issue with search seems to remain.

If you notice any other issues on this site, please let me know. I'll try to get them fixed asap.

Update: I entered an issue for my search problem in Roller's JIRA. I also managed to figure out that the problem is due to the old version of oscache that's needed by the JSPWiki plugin. Hopefully we can get the plugin upgraded to avoid this issue for other users.

Fixing XSS in JSP 2

Way back in 2007, I wrote about Java Web Frameworks and XSS. My main point was that JSP EL doesn't bother to handle XSS.

Of course, the whole problem with JSP EL could be solved if Tomcat (and other containers) would allow a flag to turn on XML escaping by default. IMO, it's badly needed to make JSP-based webapps safe from XSS.

A couple months later, I proposed a Tomcat enhancement to escape JSP's EL by default. I also entered an enhancement request for this feature and attached a patch. That issue has remained open and unfixed for 3 and 1/2 years.

Yesterday, Chin Huang posted a handy-dandy ELResolver that XML-escapes EL values.

I tried Chin's resolver in AppFuse today and it works as well as advertised. To do this, I copied his EscapeXML*.java files into my project, changed the JSP API's Maven coordinates from javax.servlet:jsp-api:2.0 to javax.servlet.jsp:jsp-api:2.1 and added the listener to web.xml.

With Struts 2 and Spring MVC, I was previously able to have ${param.xss} and pass in ?xss=<script>alert('gotcha')</script> and it would show a JavaScript alert. After using Chin's ELResolver, it prints the string on the page instead of displaying an alert.

Thanks to Chin Huang for this patch! If you're using JSP, I highly recommend you add this to your projects as well.

Implementing Ajax Authentication using jQuery, Spring Security and HTTPS

I've always had a keen interest in implementing security in webapps. I implemented container-managed authentication (CMA) in AppFuse in 2002, watched Tomcat improve it's implementation in 2003 and implemented Remember Me with CMA in 2004. In 2005, I switched from CMA to Acegi Security (now Spring Security) and never looked back. I've been very happy with Spring Security over the years, but also hope to learn more about Apache Shiro and implementing OAuth to protect JavaScript APIs in the near future.

I was recently re-inspired to learn more about security when working on a new feature at Overstock.com. The feature hasn't been released yet, but basically boils down to allowing users to login without leaving a page. For example, if they want to leave a review on a product, they would click a link, be prompted to login, enter their credentials, then continue to leave their review. The login prompt and subsequent review would likely be implemented using a lightbox. While lightboxes are often seen in webapps these days because they look good, it's also possible Lightbox UIs provide a poor user experience. User experience aside, I think it's interesting to see what's required to implement such a feature.

To demonstrate how we did it, I whipped up an example using AppFuse Light, jQuery and Spring Security. The source is available in my ajax-login project on GitHub. To begin, I wanted to accomplish a number of things to replicate the Overstock environment:

1. Force HTTPS for authentication.
2. Allow testing HTTPS without installing a certificate locally.
3. Implement a RESTful LoginService that allows users to login.
4. Implement login with Ajax, with the request coming from an insecure page.

Forcing HTTPS with Spring Security
The first feature was fairly easy to implement thanks to Spring Security. Its configuration supports a requires-channel attribute that can be used for this. I used this to force HTTPS on the "users" page and it subsequently causes the login to be secure.

<intercept-url pattern="/app/users" access="ROLE_ADMIN" requires-channel="https"/>

Testing HTTPS without adding a certificate locally
After making the above change in security.xml, I had to modify my jWebUnit test to work with SSL. In reality, I didn't have to modify the test, I just had to modify the configuration that ran the test. In my last post, I wrote about adding my 'untrusted' cert to my JVM keystore. For some reason, this works for HttpClient, but not for jWebUnit/HtmlUnit. The good news is I figured out an easier solution - adding the trustStore and trustStore password as system properties to the maven-failsafe-plugin configuration.

<artifactId>maven-failsafe-plugin</artifactId>
<version>2.7.2</version>
<configuration>
    <includes>
        <include>**/*WebTest.java</include>
    </includes>
    <systemPropertyVariables>
      <javax.net.ssl.trustStore>${project.build.directory}/ssl.keystore</javax.net.ssl.trustStore>
      <javax.net.ssl.trustStorePassword>appfuse</javax.net.ssl.trustStorePassword>
    </systemPropertyVariables>
</configuration>

The disadvantage to doing things this way is you'll have to pass these in as arguments when running unit tests in your IDE.

Implementing a LoginService
Next, I set about implementing a LoginService as a Spring MVC Controller that returns JSON thanks to the @ResponseBody annotation and Jackson.

package org.appfuse.examples.web;

import org.appfuse.model.User;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.security.authentication.AuthenticationManager;
import org.springframework.security.authentication.BadCredentialsException;
import org.springframework.security.authentication.UsernamePasswordAuthenticationToken;
import org.springframework.security.core.Authentication;
import org.springframework.security.core.context.SecurityContextHolder;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.ResponseBody;

@Controller
@RequestMapping("/api/login.json")
public class LoginService {

  @Autowired
  @Qualifier("authenticationManager")
  AuthenticationManager authenticationManager;

  @RequestMapping(method = RequestMethod.GET)
  @ResponseBody
  public LoginStatus getStatus() {
    Authentication auth = SecurityContextHolder.getContext().getAuthentication();
    if (auth != null && !auth.getName().equals("anonymousUser") && auth.isAuthenticated()) {
      return new LoginStatus(true, auth.getName());
    } else {
      return new LoginStatus(false, null);
    }
  }

  @RequestMapping(method = RequestMethod.POST)
  @ResponseBody
  public LoginStatus login(@RequestParam("j_username") String username,
                           @RequestParam("j_password") String password) {

    UsernamePasswordAuthenticationToken token = new UsernamePasswordAuthenticationToken(username, password);
    User details = new User(username);
    token.setDetails(details);

    try {
      Authentication auth = authenticationManager.authenticate(token);
      SecurityContextHolder.getContext().setAuthentication(auth);
      return new LoginStatus(auth.isAuthenticated(), auth.getName());
    } catch (BadCredentialsException e) {
      return new LoginStatus(false, null);
    }
  }

  public class LoginStatus {

    private final boolean loggedIn;
    private final String username;

    public LoginStatus(boolean loggedIn, String username) {
      this.loggedIn = loggedIn;
      this.username = username;
    }

    public boolean isLoggedIn() {
      return loggedIn;
    }

    public String getUsername() {
      return username;
    }
  }
}

To verify this class worked as expected, I wrote a unit test using JUnit and Mockito. I used Mockito because Overstock is transitioning to it from EasyMock and I've found it very simple to use.

package org.appfuse.examples.web;

import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import org.mockito.Matchers;
import org.springframework.security.authentication.AuthenticationManager;
import org.springframework.security.authentication.BadCredentialsException;
import org.springframework.security.authentication.TestingAuthenticationToken;
import org.springframework.security.core.Authentication;
import org.springframework.security.core.context.SecurityContext;
import org.springframework.security.core.context.SecurityContextHolder;
import org.springframework.security.core.context.SecurityContextImpl;

import static org.junit.Assert.*;
import static org.mockito.Mockito.*;

public class LoginServiceTest {

  LoginService loginService;
  AuthenticationManager authenticationManager;

  @Before
  public void before() {
    loginService = new LoginService();
    authenticationManager = mock(AuthenticationManager.class);
    loginService.authenticationManager = authenticationManager;
  }

  @After
  public void after() {
    SecurityContextHolder.clearContext();
  }

  @Test
  public void testLoginStatusSuccess() {
    Authentication auth = new TestingAuthenticationToken("foo", "bar");
    auth.setAuthenticated(true);
    SecurityContext context = new SecurityContextImpl();
    context.setAuthentication(auth);
    SecurityContextHolder.setContext(context);

    LoginService.LoginStatus status = loginService.getStatus();
    assertTrue(status.isLoggedIn());
  }

  @Test
  public void testLoginStatusFailure() {
    LoginService.LoginStatus status = loginService.getStatus();
    assertFalse(status.isLoggedIn());
  }

  @Test
  public void testGoodLogin() {
    Authentication auth = new TestingAuthenticationToken("foo", "bar");
    auth.setAuthenticated(true);
    when(authenticationManager.authenticate(Matchers.<Authentication>anyObject())).thenReturn(auth);
    LoginService.LoginStatus status = loginService.login("foo", "bar");
    assertTrue(status.isLoggedIn());
    assertEquals("foo", status.getUsername());
  }

  @Test
  public void testBadLogin() {
    Authentication auth = new TestingAuthenticationToken("foo", "bar");
    auth.setAuthenticated(false);
    when(authenticationManager.authenticate(Matchers.anyObject()))
        .thenThrow(new BadCredentialsException("Bad Credentials"));
    LoginService.LoginStatus status = loginService.login("foo", "bar");
    assertFalse(status.isLoggedIn());
    assertEquals(null, status.getUsername());
  }
}

Implement login with Ajax
The last feature was the hardest to implement and still isn't fully working as I'd hoped. I used jQuery and jQuery UI to implement a dialog that opens the login page on the same page rather than redirecting to the login page. The "#demo" locator refers to a button in the page.

Passing in the "ajax=true" parameter disables SiteMesh decoration on the login page, something that's described in my Ajaxified Body article.

var dialog = $('<div></div>');

$(document).ready(function() {
    $.get('/login?ajax=true', function(data) {
        dialog.html(data);
        dialog.dialog({
            autoOpen: false,
	       title: 'Authentication Required'
        });
    });

    $('#demo').click(function() {
      dialog.dialog('open');
      // prevent the default action, e.g., following a link
      return false;
    });
});

Instead of adding a click handler to a specific id, it's probably better to use a CSS class that indicates authentication is required for a link, or -- even better -- use Ajax to see if the link is secured.

The login page then has the following JavaScript to add a click handler to the "login" button that submits the request securely to the LoginService.

var getHost = function() {
    var port = (window.location.port == "8080") ? ":8443" : "";
    return ((secure) ? 'https://' : 'http://') + window.location.hostname + port;
};

var loginFailed = function(data, status) {
    $(".error").remove();
    $('#username-label').before('<div class="error">Login failed, please try again.</div>');
};

$("#login").live('click', function(e) {
    e.preventDefault();
    $.ajax({url: getHost() + "/api/login.json",
        type: "POST",
        data: $("#loginForm").serialize(),
        success: function(data, status) {
            if (data.loggedIn) {
                // success
                dialog.dialog('close');
                location.href= getHost() + '/users';
            } else {
                loginFailed(data);
            }
        },
        error: loginFailed
    });
});

The biggest secret to making this all work (the HTTP -> HTTPS communication, which is considered cross-domain), is the window.name Transport and the jQuery plugin that implements it. To make this plugin work with Firefox 3.6, I had to implement a Filter that adds Access-Control headers. A question on Stackoverflow helped me figure this out.

public class OptionsHeadersFilter implements Filter {

    public void doFilter(ServletRequest req, ServletResponse res, FilterChain chain)
            throws IOException, ServletException {
        HttpServletResponse response = (HttpServletResponse) res;

        response.setHeader("Access-Control-Allow-Origin", "*");
        response.setHeader("Access-Control-Allow-Methods", "GET,POST");
        response.setHeader("Access-Control-Max-Age", "360");
        response.setHeader("Access-Control-Allow-Headers", "x-requested-with");

        chain.doFilter(req, res);
    }

    public void init(FilterConfig filterConfig) {
    }

    public void destroy() {
    }
}

Issues
I encountered a number of issues when implementing this in the ajax-login project.

• If you try to run this with ports (e.g. 8080 and 8443) in your URLs, you'll get a 501 (Not Implemented) response. Removing the ports by fronting with Apache and mod_proxy solves this problem.
• If you haven't accepted the certificate in your browser, the Ajax request will fail. In the example, I solved this by clicking on the "Users" tab to make a secure request, then going back to the homepage to try and login.
• The jQuery window.name version 0.9.1 doesn't work with jQuery 1.5.0. The error is "$.httpSuccess function not found."
• Finally, even though I was able to authenticate successfully, I was unable to make the authentication persist. I tried adding the following to persist the updated SecurityContext to the session, but it doesn't work. I expect the solution is to create a secure JSESSIONID cookie somehow.

  @Autowired
  SecurityContextRepository repository;
  
  @RequestMapping(method = RequestMethod.POST)
  @ResponseBody
  public LoginStatus login(@RequestParam("j_username") String username,
                           @RequestParam("j_password") String password,
                           HttpServletRequest request, HttpServletResponse response) {
  
      UsernamePasswordAuthenticationToken token = new UsernamePasswordAuthenticationToken(username, password);
      ...
  
      try {
          Authentication auth = authenticationManager.authenticate(token);
          SecurityContextHolder.getContext().setAuthentication(auth);
          // save the updated context to the session
          repository.saveContext(SecurityContextHolder.getContext(), request, response);
          return new LoginStatus(auth.isAuthenticated(), auth.getName());
      } catch (BadCredentialsException e) {
          return new LoginStatus(false, null);
      }
  }
  

Conclusion
This article has shown you how to force HTTPS for login, how to do integration testing with a self-generated certificate, how to implement a LoginService with Spring MVC and Spring Security, as well as how to use jQuery to talk to a service cross-domain with the window.name Transport. While I don't have everything working as much as I'd like, I hope this helps you implement a similar feature in your applications.

One thing to be aware of is with lightbox/dialog logins and HTTP -> HTTPS is that users won't see a secure icon in their address bar. If your app has sensitive data, you might want to force https for your entire app. OWASP's Secure Login Pages has a lot of good tips in this area.

Update: I've posted a demo of the ajax-login webapp. Thanks to Contegix for hosting the demo and helping obtain/install an SSL certificate so quickly.

Integration Testing with HTTP, HTTPS and Maven

Earlier this week, I was tasked with getting automated integration tests working in my project at Overstock.com. By automated, I mean that ability to run "mvn install" and have the following process cycled through:

• Start a container
• Deploy the application
• Run all integration tests
• Stop the container

Since it makes sense for integration tests to run in Maven's integration-test phase, I first configured the maven-surefire-plugin to skip tests in the test phase and execute them in the integration-test phase. I used the <id>default-phase</id> syntax to override the plugins' usual behavior.

<plugin>
  <artifactId>maven-surefire-plugin</artifactId>
  <executions>
    <execution>
      <id>default-test</id>
      <configuration>
        <excludes>
          <exclude>**/*Test*.java</exclude>
        </excludes>
      </configuration>
    </execution>
    <execution>
      <id>default-integration-test</id>
      <phase>integration-test</phase>
      <goals>
        <goal>test</goal>
      </goals>
      <configuration>
        <includes>
          <include>**/*Test.java</include>
        </includes>
        <excludes>
          <exclude>none</exclude>
          <exclude>**/TestCase.java</exclude>
        </excludes>
      </configuration>
    </execution>
  </executions>
</plugin>

After I had this working, I moved onto getting the container started and stopped properly. In the past, I've done this using Cargo and it's always worked well for me. Apart from the usual setup I use in AppFuse archetypes (example pom.xml), I added a couple additional items:

• Added <timeout>180000</timeout> so the container would wait up to 3 minutes for the WAR to deploy.
• In configuration/properties, specified <context.path>ROOT</context.path> so the app would deploy at the / context path.
• In configuration/properties, specified <cargo.protocol>https</cargo.protocol> since many existing unit tests made requests to secure resources.

I started by using Cargo with Tomcat and had to create certificate keystore in order to get Tomcat to start with SSL enabled. After getting it to start, I found the tests failed with the following errors in the logs:

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: 
PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: 
unable to find valid certification path to requested target
	at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1649)

Co-workers told me this was easily solved by adding my 'untrusted' cert to my JVM keystore. Once all this was working, I thought I was good to go, but found that some tests were still failing. The failures turned out to be because they were talking to http and https was the only protocol enabled. After doing some research, I discovered that Cargo doesn't support starting on both http and https ports.

So back to the drawing board I went. I ended up turning to the maven-jetty-plugin and the tomcat-maven-plugin to get the functionality I was looking for. I also automated the certificate keystore generation using the keytool-maven-plugin. Below is the extremely-verbose 95-line profiles section of my pom.xml that allows either container to be used.

Sidenote: I wonder how this same setup would look using Gradle?

<profiles>
  <profile>
    <id>jetty</id>
    <activation>
      <activeByDefault>true</activeByDefault>
    </activation>
    <build>
      <plugins>
        <plugin>
          <groupId>org.mortbay.jetty</groupId>
          <artifactId>maven-jetty-plugin</artifactId>
          <version>6.1.26</version>
          <configuration>
            <contextPath>/</contextPath>
            <connectors>
              <connector implementation="org.mortbay.jetty.nio.SelectChannelConnector">
                <!-- forwarded == true interprets x-forwarded-* headers -->
                <!-- http://docs.codehaus.org/display/JETTY/Configuring+mod_proxy -->
                <forwarded>true</forwarded>
                <port>8080</port>
                <maxIdleTime>60000</maxIdleTime>
              </connector>
              <connector implementation="org.mortbay.jetty.security.SslSocketConnector">
                <forwarded>true</forwarded>
                <port>8443</port>
                <maxIdleTime>60000</maxIdleTime>
                <keystore>${project.build.directory}/ssl.keystore</keystore>
                <password>overstock</password>
                <keyPassword>overstock</keyPassword>
              </connector>
            </connectors>
            <stopKey>overstock</stopKey>
            <stopPort>9999</stopPort>
          </configuration>
          <executions>
            <execution>
              <id>start-jetty</id>
              <phase>pre-integration-test</phase>
              <goals>
                <goal>run-war</goal>
              </goals>
              <configuration>
                <daemon>true</daemon>
              </configuration>
            </execution>
            <execution>
              <id>stop-jetty</id>
              <phase>post-integration-test</phase>
              <goals>
                <goal>stop</goal>
              </goals>
            </execution>
          </executions>
        </plugin>
      </plugins>
    </build>
  </profile>
  <profile>
    <id>tomcat</id>
    <build>
      <plugins>
        <plugin>
          <groupId>org.codehaus.mojo</groupId>
          <artifactId>tomcat-maven-plugin</artifactId>
          <version>1.1</version>
          <configuration>
            <addContextWarDependencies>true</addContextWarDependencies>
            <fork>true</fork>
            <path>/</path>
            <port>8080</port>
            <httpsPort>8443</httpsPort>
            <keystoreFile>${project.build.directory}/ssl.keystore</keystoreFile>
            <keystorePass>overstock</keystorePass>
          </configuration>
          <executions>
            <execution>
              <id>start-tomcat</id>
              <phase>pre-integration-test</phase>
              <goals>
                <goal>run-war</goal>
              </goals>
            </execution>
            <execution>
              <id>stop-tomcat</id>
              <phase>post-integration-test</phase>
              <goals>
                <goal>shutdown</goal>
              </goals>
            </execution>
          </executions>
        </plugin>
      </plugins>
    </build>
  </profile>
</profiles>

With this setup in place, I was able to automate running our integration tests by simply typing "mvn install" (for Jetty) or "mvn install -Ptomcat" (for Tomcat). For running in Hudson, it's possible I'll have to further enhance things to randomize the port and pass that into tests as a system property. The build-helper-maven-plugin and its reserve-network-port goal is a nice way to do this. Note: if you want to run more than one instance of Tomcat at a time, you might have to randomize the ajp and rmi ports to avoid collisions.

The final thing I encountered was our app didn't shutdown gracefully. Luckily, this was fixed in a newer version of our core framework and upgrading fixed the problem. Here's the explanation from an architect on the core framework team.

The hanging problem was caused by the way the framework internally aggregated statistics related to database connection usage and page response times. The aggregation runs on a separate thread but not as a daemon thread. Previously, the aggregation threads weren't being terminated on shutdown so the JVM would hang waiting for them to finish. In the new frameworks, the aggregation threads are terminated on shutdown.

Hopefully this post helps you test your secure and unsecure applications at the same time. At the same time, I'm hoping it motivates the Cargo developers to add simultaneous http and https support.

Update: In the comments, Ron Piterman recommended I use the Maven Failsafe Plugin because its designed to run integration tests while Surefire Plugin is for unit tests. I changed my configuration to the following and everything still passes. Thanks Ron!

<plugin>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>2.7.2</version>
  <configuration>
    <skipTests>true</skipTests>
  </configuration>
</plugin>
<plugin>
  <artifactId>maven-failsafe-plugin</artifactId>
  <version>2.7.2</version>
  <configuration>
    <includes>
      <include>**/*Test.java</include>
    </includes>
    <excludes>
      <exclude>**/TestCase.java</exclude>
    </excludes>
  </configuration>
  <executions>
    <execution>
      <id>integration-test</id>
      <phase>integration-test</phase>
      <goals>
        <goal>integration-test</goal>
      </goals>
    </execution>
    <execution>
      <id>verify</id>
      <phase>verify</phase>
      <goals>
        <goal>verify</goal>
      </goals>
    </execution>
  </executions>
</plugin>

Update 2: In addition to application changes to solve hanging issues, I also had to change my Jetty Plugin configuration to use a different SSL connector implementation. This also required adding the jetty-sslengine dependency, which has been renamed to jetty-ssl for Jetty 7.

<connector implementation="org.mortbay.jetty.security.SslSelectChannelConnector">
...
<dependencies>
  <dependency>
    <groupId>org.mortbay.jetty</groupId>
    <artifactId>jetty-sslengine</artifactId>
    <version>6.1.26</version>
  </dependency>
</dependencies>

Implementing Extensionless URLs with Tapestry, Spring MVC, Struts 2 and JSF

For the past couple of weeks, I've spent several evening hours implementing extensionless URLs in AppFuse. I've been wanting to do this ever since I wrote about how to do it a few years ago. This article details my experience and will hopefully help others implement this feature in their webapps.

First of all, I used the UrlRewriteFilter, one of my favorite Java open source projects. Then I followed a pattern I found in Spring's "mvc-basic" sample app from MVC Simplifications in Spring 3.0. The app has since changed (because SpringSource integrated UrlRewriteFilter-type functionality in Spring MVC), but the pattern was basically path-matching instead of extension-mapping. That is, the "dispatcher" for the web framework was mapped to /app/* instead of *.html.

Prior to the move to extensionless URLs, AppFuse used *.html for its mapping and this seemed to cause users problems when they wanted to serve up static HTML files. To begin with, I removed all extensions from URLs in tests (Canoo WebTest is used for testing the UI). I also did this for any links in the view pages and redirects in the Java code. This provided a decent foundation to verify my changes worked. Below are details about each framework I did this for, starting with the one that was easiest and moving to hardest.

Tapestry 5
Tapestry was by far the easiest to integrate extensionless URLs into. This is because it's a native feature of the framework and was already integrated as part of Serge Eby's Tapestry 5 implementation. In the end, the only things I had to do where 1) add a couple entries for CXF (mapped to /services/*) and DWR (/dwr/*) to my urlrewrite.xml and 2) change the UrlRewriteFilter so it was only mapped to REQUEST instead of both REQUEST and FORWARD. Below are the mappings I added for CXF and DWR.

<urlrewrite default-match-type="wildcard">
    ...
    <rule>
        <from>/dwr/**</from>
        <to>/dwr/$1</to>
    </rule>
    <rule>
        <from>/services/**</from>
        <to>/services/$1</to>
    </rule>
</urlrewrite>

Spring MVC
I had a fair amount of experience with Spring MVC and extensionless URLs. Both the Spring MVC applications we developed last year at Time Warner Cable used them. To change from a *.html mapping to /app/* was pretty easy and involved removing more code than I added. Previously, I had a StaticFilter that looked for HTML files and if it didn't find them, it dispatched to Spring's DispatcherServlet. I was able to remove this class and make the web.xml file quite a bit cleaner.

To make UrlRewriteFilter and Spring Security play well together, I had to move the securityFilter so it came after the rewriteFilter and add an INCLUDE dispatcher so included JSPs would have a security context available to them.

<filter-mapping>
    <filter-name>rewriteFilter</filter-name>
    <url-pattern>/*</url-pattern>
</filter-mapping>
<filter-mapping>
    <filter-name>securityFilter</filter-name>
    <url-pattern>/*</url-pattern>
    <dispatcher>REQUEST</dispatcher>
    <dispatcher>FORWARD</dispatcher>
    <dispatcher>INCLUDE</dispatcher>
</filter-mapping>

The only other things I had to change were security.xml and dispatcher-servlet.xml to remove the .html extensions. The urlrewrite.xml file was fairly straightforward. I used the following at the bottom as a catch-all for dispatching to Spring MVC.

<rule>
    <from>/**</from>
    <to>/app/$1</to>
</rule>
<outbound-rule>
    <from>/app/**</from>
    <to>/$1</to>
</outbound-rule>

Then I added a number of other rules for j_security_check, DWR, CXF and static assets (/images, /scripts, /styles, /favicon.ico). You can view the current urlrewrite.xml in FishEye. The only major issue I ran into was that Spring Security recorded protected URLs as /app/URL so I had to add a rule to redirect when this happened after logging in.

<rule>
    <from>/app/**</from>
    <to last="true" type="redirect">%{context-path}/$1</to>
</rule>

Struts 2
Using extensionless URLs with Struts 2 is likely pretty easy thanks to the Convention Plugin. Even though this plugin is included in AppFuse, it's not configured with the proper constants and I have struts.convention.action.disableScanning=true in struts.xml. It looks like I had to do this when I upgraded from Struts 2.0.x to Struts 2.1.6. It's true AppFuse's Struts 2 support could use a bit of love to be aligned with Struts 2's recommended practices, but I didn't want to spend the time doing it as part of this exercise.

With Struts 2, I tried the path-mapping like I did with Spring MVC, but ran into issues. Instead, I opted to use an ".action" extension by changing struts.action.extension from "html" to "action," in struts.xml. Then I had to do a bunch of filter re-ordering and dispatcher changes. Before, with a .html extension, I had all filters mapped to /* and in the following order.

Similar to Spring MVC, I had to remove the rewriteFilter in front of the securityFilter and I was able to remove the staticFilter. I also had to map the struts filter to *.action instead of /* to stop Struts from trying to catch static asset and DWR/CXF requests. Below is the order of filters and their dispatchers that seems to work best.

From there, it was a matter of modifying urlrewrite.xml to have the following catch-all and rules for static assets, j_security_check and DWR/CXF.

<rule match-type="regex">
    <from>^([^?]*)/([^?/\.]+)(\?.*)?$</from>
    <to last="true">$1/$2.action$3</to>
</rule>
<outbound-rule match-type="regex">
    <from>^(.*)\.action(\?.*)?$</from>
    <to last="false">$1$2</to>
</outbound-rule>

JSF
JSF was by far the most difficult to get extensionless URLs working with. I'm not convinced it's impossible, but I spent a several hours over a few days and was unsuccessful in completely removing them. I was able to make things work so I could request pages without an extension, but found when clicking buttons and links, the extension would often show up in the URL. I'm also still using JSF 1.2, so it's possible that upgrading to 2.0 would solve many of the issues I encountered.

For the time being, I've changed my FacesServlet mapping from *.html to *.jsf. As with Struts, I had issues when I tried to map it to /app/*. Other changes include changing the order of dispatchers and filters, the good ol' catch-all in urlrewrite.xml and modifying security.xml. For some reason, I wasn't able to get file upload working without adding an exception to the outbound-rule.

<rule match-type="regex">
    <from>^([^?]*)/([^?/\.]+)(\?.*)?$</from>
    <to last="true">$1/$2.jsf</to>
</rule>
<outbound-rule match-type="regex">
  <!-- TODO: Figure out how to make file upload work w/o using *.jsf -->
    <condition type="path-info">selectFile</condition>
    <from>^(.*)\.jsf(\?.*)?$</from>
    <to last="false">$1$2</to>
</outbound-rule>

I also spent a couple hours trying to get Pretty Faces to work. I wrote about my issues on the forums. I tried writing a custom Processor to strip the extension, but found that I'd get into an infinite loop where the processor kept getting called. To workaround this, I tried using Spring's RequestContextHolder to ensure the processor only got invoked once, but that proved fruitless. Finally, I tried inbound and outbound custom processors, but failed to get those working. The final thing I tried was url-mappings for each page in pretty-config.xml.

<url-mapping>
  <pattern value="/admin/users"/>
  <view-id value="/admin/users.jsf"/>
</url-mapping>
<url-mapping>
  <pattern value="/mainMenu"/>
  <view-id value="/mainMenu.jsf"/>
</url-mapping>

The issue with doing this was that some of the navigation rules in my faces-config.xml stopped working. I didn't spend much time trying to diagnose the problem because I didn't like having to add an entry for each page in the application. The one nice thing about Pretty Faces is it did allow me to do things like the following, which I formerly did with a form that auto-submitted when the page loaded.

<url-mapping>
  <pattern value="/passwordHint/#{username}"/>
  <view-id value="/passwordHint.jsf"/>
  <action>#{passwordHint.execute}</action>
</url-mapping>

Conclusion
My journey implementing extensionless URLs was an interesting one, and I solidified my knowledge about ordering of filters, dispatchers and the UrlRewriteFilter. I still think I have more to learn about properly implementing extensionless URLs in Struts 2 and JSF and I hope to do that in the near future. I believe Struts' Convention Plugin will help me and JSF 2 + Pretty Faces will hopefully work nicely too. Of course, it'd be great if all Java Web Frameworks had an easy mechanism for producing and consuming extensionless URLs. In the meantime, thank goodness for the UrlRewriteFilter.

If you'd like to try AppFuse and its shiny new URLs, see the QuickStart Guide and choose the 2.1.0-SNAPSHOT version.

Making Code Generation Smarter with Maven

As you might've read in my last entry, I recently started a new gig with Overstock.com. On my first day, I was quickly immersed into the development process by joining the Conversion Team. The Conversion Team is responsible for developing the checkout UI and handling payments from customers. I quickly discovered Overstock was mostly a Linux + Eclipse Shop and did my best to get my favorite Mac + IntelliJ + JRebel installed and configured. Thanks to my new Team Lead, I was able to get everything up and running the first day, as well as checkin my first contribution: making mvn jetty:run work so I didn't have to use my IDE to deploy to Tomcat.

In setting up my environment, I couldn't help but notice running jetty:run took quite a while to run each time. Specifically, the build process took 45 seconds to start executing the Jetty plugin, then another 23 seconds to startup after that. The first suspicious thing I noticed was that the UI templates were being re-generated and compiled on each execution. The UI Templating Framework at Overstock is Jamon, and is described as follows:

Jamon is a text template engine for Java, useful for generating dynamic HTML, XML, or any text-based content. In a typical Model-View-Controller architecture, Jamon clearly is aimed at the View (or presentation) layer.

Because it is compiled to non-reflective Java code, and statically type-checked, Jamon is ideally suited to support refactoring of template-based UI applications. Using mock objects -like functionality, Jamon also facilitates unit testing of the controller and view.

To generate .java files from .jamon templates, we use the Jamon Plugin for Maven. Remembering that the Maven Compiler Plugin has an incremental-compile feature, I turned to its source code to find out how to implement this in the Jamon plugin. I was pleasantly surprised to find the StaleSourceScanner. This class allows you to easily compare two files to see if the source needs to re-examined for generation or compilation.

I noticed the Jamon Plugin had the following code to figure out which files it should generate into .java files:

private List<File> accumulateSources(File p_templateSourceDir)
{
  final List<File> result = new ArrayList<File>();
  if (p_templateSourceDir == null)
  {
    return result;
  }
  for (File f : p_templateSourceDir.listFiles())
  {
    if (f.isDirectory())
    {
      result.addAll(accumulateSources(f));
    }
    else if (f.getName().toLowerCase(Locale.US).endsWith(".jamon"))
    {
      String filePath = f.getPath();
       // FIXME !?
      String basePath = templateSourceDir().getAbsoluteFile().toString();
      result.add(new File(filePath.substring(basePath.length() + 1)));
    }
  }
  return result;
}

I changed it to be smarter and only generate changed templates with the following code:

private List<File> accumulateSources(File p_templateSourceDir) throws MojoExecutionException
{
  final List<File> result = new ArrayList<File>();
  if (p_templateSourceDir == null)
  {
    return result;
  }
  SourceInclusionScanner scanner = getSourceInclusionScanner( staleMillis );
  SourceMapping mapping = new SuffixMapping( ".jamon", ".java");

  scanner.addSourceMapping( mapping );

  final Set<File> staleFiles = new LinkedHashSet<File>();

  for (File f : p_templateSourceDir.listFiles())
  {
    if (!f.isDirectory())
    {
      continue;
    }

    try
    {
      staleFiles.addAll( scanner.getIncludedSources(f.getParentFile(), templateOutputDir()));
    }
    catch ( InclusionScanException e )
    {
      throw new MojoExecutionException(
        "Error scanning source root: \'" + p_templateSourceDir.getPath()
          + "\' " + "for stale files to recompile.", e );
    }
  }

  // Trim root path from file paths
  for (File file : staleFiles) {
    String filePath = file.getPath();
    String basePath = templateSourceDir().getAbsoluteFile().toString();
    result.add(new File(filePath.substring(basePath.length() + 1)));
  }
}

This method references a getSourceInclusionScanner() method, which is implemented as follows:

protected SourceInclusionScanner getSourceInclusionScanner( int staleMillis )
{
  SourceInclusionScanner scanner;

  if ( includes.isEmpty() && excludes.isEmpty() )
  {
      scanner = new StaleSourceScanner( staleMillis );
  }
  else
  {
      if ( includes.isEmpty() )
      {
          includes.add( "**/*.jamon" );
      }
      scanner = new StaleSourceScanner( staleMillis, includes, excludes );
  }

  return scanner;
}

If you're using Jamon and its Maven Plugin, you can view my patch at SourceForge. If you're looking to include this functionality in your project, I invite you to look at the code I learned from in the Maven Compiler's AbstractCompilerMojo class.

After making this change, I was able to reduce the build execution time by over 50%. Now it takes 20 seconds to hit the Jetty plugin and 42 seconds to finishing starting. Of course, in an ideal world, I'd like to get this down to 20 seconds or less. Strangely enough, the easiest way to do this seems to be simple: use Linux.

On the Linux desktop they provided me, it takes 12 seconds to hit the Jetty plugin and 23 seconds to finish starting. I'd like to think this is a hardware thing, but it only get 20% faster on OS X when using an 8GB RAM + SSD machine (vs. a 4GB + 5400 drive). Since Overstock has provided me with a 4GB MacBook Pro, I'm considering installing Ubuntu on it, just to see what the difference is.

In related news, Overstock.com is looking to hire a whole bunch of Java Developers this year. The pictures of the new Provo office look pretty sweet. Of course, you can also work at HQ, which is a mere 25 minutes from some of the best skiing in the world. Personally, I think Colorado's powder is better, but I can't argue with the convenience of no traffic. In addition to full-time gigs, they've started hiring more remote contractors like myself, so they pretty much have something for everyone. So if you love Java, like to get some turns in before work, and aren't an asshole - you should and I'll try to hook you up.

Update: After writing this post, I received an email from Neil Hartner questioning these numbers. Basically, he was able to get his MacBook Pro to run just as fast as Linux. Turns out, the reason my Mac was so much slower was because JRebel was configured in my MAVEN_OPTS. JRebel's FAQ does state the following:

Does JRebel make the server start up slower?
JRebel needs to do more work on startup (search more places for classes and resources, instrument classes, etc), so some slowdown can be expected. If it's larger than 50% please contact [email protected].

Since it's right around 50% slower, I guess there's no reason to call them. My guess is the best thing to do is remove JRebel from MAVEN_OPTS, but have an alias that can enable it, or simply run it from your IDE.

How I Calculated Ratings for My JVM Web Frameworks Comparison

When I re-wrote my Comparing JVM Web Frameworks presentation from scratch, I decided to add a matrix that allows you to rate a framework based on 20 different criteria. The reason I did this was because I'd used this method when choosing an Ajax framework for Evite last year. The matrix seemed to work well for selecting the top 5 frameworks, but it also inspired a lot of discussion in the community that my ratings were wrong.

I expected this, as I certainly don't know every framework as well as I'd like. The mistake I made was asking for the community to provide feedback on my ratings without describing how I arrived at them. From Peter Thomas's blog:

What you are doing is adjusting ratings based on who in the community shouts the loudest. I can't help saying that this approach comes across as highly arrogant and condescending, you seem to expect framework developers and proponents to rush over and fawn over you to get better ratings, like waiters in a restaurant trying to impress a food-critic for Michelin stars.

I apologize for giving this impression. It certainly wasn't my intent. By having simple numbers (1.0 == framework does well, 0.5 == framework is OK and 0 == framework not good at criteria) with no rationalization, I can see how the matrix can be interpreted as useless (or to put it bluntly, as something you should wipe your ass with). I don't blame folks for getting angry.

For my Rich Web Experience presentation, I documented why I gave each framework the rating I did. Hopefully this will allow folks to critique my ratings more constructively and I can make the numbers more accurate. You can view this document below or on Google Docs.

In the end, what I was hoping to do with this matrix was to simply highlight a technique for choosing a web framework. Furthermore, I think adding a "weight" to each criteria is important because things like books often aren't as important as REST support. To show how this might be done, I added a second sheet to the matrix and made up some weighting numbers. I'd expect anyone that wants to use this to downloaded the matrix, verify the ratings are accurate for your beliefs and weight the criteria accordingly.

Of course, as I and many others have said, the best way to choose a web framework is to try them yourself. I emphasized this at the end of my presentation with the following two slides.

RE: Moving from Spring to Java EE 6: The Age of Frameworks is Over

Last Tuesday, Cameron McKenzie wrote an interesting article on TheServerSide titled Moving from Spring to Java EE 6: The Age of Frameworks is Over. In this article, Cameron says the following:

J2EE represents the past, and Java EE 6 represents the future. Java EE 6 promises us the ability to go beyond frameworks. Frameworks like Spring are really just a bridge between the mistakes of the J2EE past and the success of the Java EE 6 future. Frameworks are out, and extensions to the Java EE 6 platform are in. Now is the time to start looking past Spring, and looking forward to Seam and Weld and CDI technologies.

He then links to an article titled Spring to Java EE - A Migration Experience, an article written by JBoss's Lincoln Baxter. In this article, Lincoln talks about many of the technologies in Java EE 6, namely JPA, EJB, JSF, CDI and JAX-RS. He highlights all the various XML files you'll need to know about and the wide variety of Java EE 6 application servers: JBoss AS 6 and GlassFish v3.

I don't have a problem with Lincoln's article, in fact I think it's very informative and some of the best documentation I've seen for Java EE 6.

I do have some issues with Cameron's statements that frameworks are mistakes of the J2EE past and that Java EE 6 represents the future. Open source frameworks made J2EE successful. Struts and Hibernate came out in the early days of J2EE and still exist today. Spring came out shortly after and has turned into the do-everything J2EE implementation it was trying to fix. Java EE 6 might be a better foundation to build upon, but it's certainly not going to replace frameworks.

To prove my point, let's start by looking at the persistence layer. We used to have Hibernate based on JDBC, now we have JPA implementations built on top of the JPA API. Is JPA a replacement for all persistence frameworks? I've worked with it and think it's a good API, but the 2.0 version isn't available in a Maven repo and Alfresco recently moved away from Hibernate (which == JPA IMO) to iBATIS for greater data access layer control and scalability. Looks like the age of frameworks isn't over for persistence frameworks.

The other areas that Java EE 6 covers that I believe frameworks will continue to excel in: EJB, CDI, JSF and JAX-RS. Personally, I don't have a problem with EJB 3 and think it's a vast improvement on EJB 2.x. I don't have an issue with CDI either, and as long as it resembles Guice for dependency injection, it works for me. However, when you get into the space I've been living in for the last couple years (high-traffic public internet sites), EJB and things like the "conversation-scope" feature of CDI don't buy you much. The way to make web application scale is to eliminate state and cache as much as possible, both of which Java EE doesn't provide much help for. In fact, to disable sessions in a servlet-container, you have to write a Filter like the following:

public class DisabledSessionFilter extends OncePerRequestFilter {

    /**
     * Filters requests to disable URL-based session identifiers.
     */
    @Override
    protected void doFilterInternal(final HttpServletRequest request,
                                    final HttpServletResponse response,
                                    final FilterChain chain)
            throws IOException, ServletException {

        HttpServletRequestWrapper wrappedRequest = new HttpServletRequestWrapper(request) {

            @Override
            public HttpSession getSession(final boolean create) {
                if (create) {
                    throw new UnsupportedOperationException("Session support disabled");
                }
                return null;
            }

            @Override
            public HttpSession getSession() {
                throw new UnsupportedOperationException("Session support disabled");
            }
        };

        // process next request in chain
        chain.doFilter(wrappedRequest, response);
    }
}

What about JAX-RS? Does it replace the need for frameworks? I like the idea of having a REST API in Java. However, its reference implementation is Jersey, which seems more like a framework than just Java EE. If you choose to use JAX-RS in your application, you still have to choose between CXF, Jersey, RESTEasy and Restlet. I compared these frameworks last year and found the Java EE implementation lacking in the features I needed.

Finally, let's talk about my-least-framework-web-framework: JSF. The main reason I don't like JSF is because of its 1.x version. JSF 1.0 was released a year before the Ajax term was coined (see timeline below). Not only did it take forever to develop as a spec, but it tried to be a client-component framework that was very stateful by default.

Now that JSF 2.0 is out, it has Ajax integrated and allows you to use GET instead of POST-for-everything. However, the only people that like Ajax integrated into their web frameworks are programmers scared of JavaScript (who probably shouldn't be developing your UI). Also, the best component development platform for the web is JavaScript. I recommend using an Ajax framework for your components if you really want a rich UI.

Sure you can use the likes of Tapestry and Wicket if you like POJO-based web development, but if you're looking to develop a webapp that's easy to maintain and understand, chances are that you'll do much better with traditional MVC frameworks like Spring MVC and Struts 2. The simplicity and popularity of Rails and Grails further emphasize that developers prefer these types of web frameworks.

Another reason I don't like JSF: there's very few developers in the wild happy with it. The major promoters of JSF are book authors, trainers, Java EE Vendors and MyFaces developers. Whenever I speak at conferences, I ask folks to raise their hands for the various web frameworks they're using. I always ask the JSF users to keep their hands up if they like it. Rarely do they stay up.

So it looks like we still need web frameworks.

Eberhard Wolff has an interesting post where he defends Spring and talks about the productivity comparisons between Spring and Java EE. He recommends using Grails or Spring Roo if you want the level of productivity that Ruby on Rails provides. That's a valid recommendation if you're building CRUD-based webapps, but I haven't developed those in quite some time. Nowadays, the apps I develop are true SOFEA apps, where the backend serves up XML or JSON and the frontend client is HTML/JavaScript/CSS, Android, iPad or Sony Blu-Ray players. On my current project, our services don't even talk to a database, they talk to a CMS via RESTful APIs. We use Spring's RestTemplate for this and HttpClient when it doesn't have the features we need. Not much in Java EE 6 for this type of communication. Sure, Jersey has a client, but it's certainly not part of the Java EE spec.

As far as getting Ruby on Rails' zero-turnaround productivity, I don't need Grails or Spring Roo, I simply use IDEA and JRebel.

Conclusion
I don't see how new features in Java EE 6 can mean the age of frameworks is over. Java SE and J2EE have always been foundations for frameworks. The Java EE 6 features are often frameworks in themselves that can be used outside of a Java EE container. Furthermore, Java EE 6 doesn't provide all the features you need to build a high-scale web app today. There's no caching, no stateless web framework that can serve up JSON and HTML and no hot-reload productivity enhancements like JRebel. Furthermore, there's real excitement in Javaland for languages like Scala, Groovy and JRuby. All of these languages have web frameworks that've made many developers happy.

Here's to the Age of Frameworks - may it live as long as the JVM!

P.S. If you'd like to hear me talk about web frameworks on the JVM, I'll be speaking at The Colorado Springs Open Source Meetup and Devoxx 2010 in the near future.

Scaling Flash Movies to match Browser Zoom Levels

Recently I was tasked with figuring out how to scale the Flash assets in the web application I'm working on. In the app, there's two different Flash assets: a Spotlight (cycles through images) and a Video Player. Before I started working on the issue, our assets would stay the same fixed size no matter what the browser zoom level. You can see this issue in action by going to Hulu or Fancast and zooming in/out (Command +/- on Mac, Control +/- on Windows). The Flash assets don't scale with the browser's text.

I found a lot of references for how to trap and handle resizing in JavaScript, so that's the initial path I took. I ended up having issues trapping the resize event in IE, as well as persisting the appropriate zoom level on page reload. Because of this, I ended up using a pure ActionScript solution that works much better. This article shows how I implemented both solutions.

Regardless of implementation, the first change I had to make was to move the height and width from the Flash asset (object/embed/JS) to its surrounding tag (<section> in our app). Then I changed the height/width to 100% on the Flash asset.

JavaScript Implementation
To allow zooming in ActionScript, I modified our main class to expose a "zoom" method to JavaScript:

ExternalInterface.addCallback("zoom", _zoom);

...

private function _zoom(scale:Number):void {
    _view.scaleX = _view.scaleX * scale;
    _view.scaleY = _view.scaleY * scale;
}

var windowHeight;
var documentHeight;

$(document).ready(function() { 
    ...
    windowHeight = $(window).height();
    documentHeight = $(document).height();

    $(window).resize(resizeWindow);
}

// Resize Flash assets when page is zoomed
function resizeWindow() {
    var newWindowHeight = $(window).height();
    var newDocumentHeight = $(document).height();
    // if document height hasn't changed, it's a browser window resize
    // event instead of a text zoom - don't change anything
    if (newDocumentHeight === documentHeight) {
        return;
    } else {
        documentHeight = newDocumentHeight;
    }
    var scale = (windowHeight / newWindowHeight); 

    var player = getFlashMovie('playerId');
    if (player && player.zoom) {
        player.zoom(scale);
    }
    var spotlight = getFlashMovie('spotlightId');
    if (spotlight && spotlight.zoom) {
        spotlight.zoom(scale);
    }

    windowHeight = newWindowHeight;
}

This seemed to work well in Firefox, Safari and Opera, but not in IE. I found this explanation about why it might not work, but I was unsuccessful in getting IE to recognize a resize/zoom event.

To fix scaling in our Spotlight asset, I used a similar solution. However, since the Spotlight didn't have all its elements in a container (they were being added directly to the stage), I had to refactor the code to add a SpotlightView (extends Sprite) that contains the bulk of the code.

Browsers persist the zoom level you've set for a site. The problem with the solution used here is it only scales up and down properly if you start from scale = 1 and revert to scale = 1 before leaving the site. If you zoom in and close your browser, when you come back the flash movies will be scale = 1 while the rest of the site is zoomed in. To solve this problem, I attempted to save the scale value in a cookie. This worked, and I was able to read the cookie in the *.as files to scale the movie correctly. However, I experienced some issues with this approach and didn't like having to delete cookies when I wanted the Flash assets to scale correctly.

ActionScript Implementation
After discovering issues with the JavaScript implementation, I did some research to see if it was possible to listen for the browser resize event in ActionScript. The Flash Fluid Layouts and Stage Resize in AS3 tutorial clued me in that the stage could listen for a resize event.

stage.addEventListener(Event.RESIZE, resizeListener); 

After adding the above line in the initialization, I added a resizeListener function that scales based on the default dimensions. It also ensures no scaling happens in full screen mode.

private function resizeListener(e:Event):void {
    // don't scale if entering full screen mode
    if (stage.displayState == StageDisplayState.FULL_SCREEN)  {
        _view.scaleX = 1;
        _view.scaleY = 1;
    } else {
        _view.scaleX = stage.stageWidth / 964;
        _view.scaleY = stage.stageHeight / 586;
    }
}

For the Spotlight asset, there are a number of different layouts (home, featured and news). The main class has a resizeListener function that scales accordingly to which layout type is being used.

private function resizeListener(e:Event):void {
    var type:String = _view.getLayoutType();

    if (type == "featured") { 
        _view.scaleX = stage.stageWidth / 958;
       _view.scaleY = stage.stageHeight / 180;
   } else if (type == "home") { 
        _view.scaleX = stage.stageWidth / 964;
        _view.scaleY = stage.stageHeight / 428;
    } else if (type == "news") {
        _view.scaleX = stage.stageWidth / 964;
        _view.scaleY = stage.stageHeight / 189;
    }
}

Because the layout type isn't set until the XML is loaded, I listen for that event in my URLLoader.

xmlLoader.addEventListener(Event.COMPLETE, resizeListener);

With the pure ActionScript implementation, the zoom level is automatically persisted. The Event.RESIZE event is fired by the Flash plugin when the page first loads if the dimensions are not the default.

That's it! Special thanks to James Ward for clueing me into scaleX and scaleY. Hopefully Hulu and Comcast can use this tutorial to scale their video players too.

Running Selenium Tests on Sauce Labs

Recently I embarked on a mission to configure my team's Selenium testing process to support multiple browsers. We use Hudson for our continuous integration server. Since our Hudson instance runs on Solaris, testing with Firefox on Solaris didn't seem like a good representation of our clients. Our browser support matrix currently looks as follows:

At first, I attempted to use Windows VMs to run Selenium tests on IE. This was a solution that didn't work too well. The major reasons it didn't work:

1. I had issues getting the Selenium Plugin for Hudson working. Upgrading the plugin to use Selenium RC 1.0.5 may solve this issue.
2. We had some unit tests that failed on Windows. I tried using the Cygpath Plugin for Hudson (which allows you to emulate a Unix environment on Windows), but failed to get it to work.
3. We quickly realized it might become a maintenance nightmare to keep all the different VMs up-to-date.

Frustrated by these issues, I turned to Sauce Labs. They have a cloud-based model that runs Selenium tests on VMs that point back to your application. They also support many different browser/OS combinations. We asked them about support for OS X and various Windows versions and they indicated that their experience shows browsers are the same across OSes.

I'm writing this article to show you how we've configured our build process to support 1) testing locally and 2) testing on Sauce Labs. In a future post, I hope to write about how to run Selenium tests concurrently for faster execution.

Running Selenium Tests Locally
We use Maven to build our project and run our Selenium tests. Our configuration is very similar to the poms referenced in Integrating Selenium with Maven 2. Basically, we have an "itest" profile that gets invoked when we pass in -Pitest. It downloads/starts Tomcat (using Cargo), deploys our WAR, starts Selenium RC (using the selenium-maven-plugin) and executes JUnit-based tests using the maven-surefire-plugin. All of this configuration is pretty standard and something I've used on many projects over the past several years.

Beyond that, we have a custom BlockJUnit4ClassRunner class that takes screenshots and captures the HTML source for tests that fail.

public class SeleniumJUnitRunner extends BlockJUnit4ClassRunner {
    public SeleniumJUnitRunner(Class<?> klass) throws InitializationError {
        super(klass);
    }

    protected Statement methodInvoker(FrameworkMethod method, Object test) {
        if (!(test instanceof AbstractSeleniumTestCase)) {
            throw new RuntimeException("Only works with AbstractSeleniumTestCase");
        }

        final AbstractSeleniumTestCase stc = ((AbstractSeleniumTestCase) test);
        stc.setDescription(describeChild(method));

        return new InvokeMethod(method, test) {
            @Override
            public void evaluate() throws Throwable {
                try {
                    super.evaluate();
                } catch (Throwable throwable) {
                    stc.takeScreenshot("FAILURE");
                    stc.captureHtmlSource("FAILURE");
                    throw throwable;
                }
            }
        };
    }
}

To use the functionality SeleniumJUnitRunner provides, we have a parent class for all our tests. This class uses the @RunWith annotation as follows:

@RunWith(SeleniumJUnitRunner.class)
public abstract class AbstractSeleniumTestCase {
    // convenience methods
}

This class looks up the Selenium RC Server, the app location and what browser to use based on system properties. If system properties are not set, it has defaults for running locally.

public static String SERVER = System.getProperty("selenium.server");
public static String APP = System.getProperty("selenium.application");
public static String BROWSER = System.getProperty("selenium.browser");

protected Selenium selenium;

@Before
public void setUp() throws Exception {
    if (SERVER == null) {
        SERVER = "localhost";
    }

    if (BROWSER == null) {
        BROWSER = "*firefox3";
    }

    if (APP == null) {
        APP = "http://localhost:9000";
    }

    selenium = new DefaultSelenium(SERVER, 4444, BROWSER, APP);
    selenium.start("captureNetworkTraffic=true");
    selenium.getEval("window.moveTo(1,1); window.resizeTo(1021,737);");
    selenium.setTimeout("60000");
}

The system properties are specified as part of the surefire-plugin's configuration. The reason we default them in the above code is so tests can be run from IDEA as well.

<artifactId>maven-surefire-plugin</artifactId>
<version>2.5</version>
<configuration>
    <systemPropertyVariables>
        <selenium.application>${selenium.application}</selenium.application>
        <selenium.browser>${selenium.browser}</selenium.browser>
        <selenium.server>${selenium.server}</selenium.server>
    </systemPropertyVariables>
</configuration>

Running Selenium Tests in the Cloud
To run tests in the cloud, you have to do a bit of setup first. If you're behind a firewall, you'll need to setup SSH tunneling so Sauce Labs can see your machine. You'll also need to setup SSH Tunneling on your Hudson server, but installing/configuring/running locally is usually a good first step. Below are the steps I used to configure Sauce Labs' SSH Tunneling on OS X.

1. Install the Python version in /opt/tools/saucelabs. If you get an error (No local packages or download links found for install) download the egg and run it with:

sudo sh setuptools-0.6c11-py2.6.egg

NOTE: If you get an error (unable to execute gcc-4.2: No such file or directory) when installing pycrypto on OS X, you'll need to install the OS X Developer Tools.

2. Create a /opt/tools/saucelabs/local.sh script with the following in it. You should change the last parameter to use your username (instead of mraible) since Sauce Labs uses unique tunnel names.

python tunnel.py {sauce.username} {sauce.key} localhost 9000:80 mraible.local

3. Start the tunnel by executing local.sh. You should see output similar to the following.

$ sh local.sh 
/System/../Python.framework/../2.6/../twisted/internet/_sslverify.py:5: DeprecationWarning: the md5 module is deprecated; use hashlib instead
 import itertools, md5
/System/../Python.framework/../2.6/../twisted/conch/ssh/keys.py:13: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
 import sha, md5
Launching tunnel ... 
Status: new
Status: booting
Status: running
Tunnel host: ec2-75-101-216-8.compute-1.amazonaws.com
Tunnel ID: 70f15fb59d2e7ebde55a6274ddfa54dd
<sshtunnel.TunnelTransport instance at 0x10217ad88> created
requesting remote forwarding for tunnel 70f15fb59d2e7ebde55a6274ddfa54dd 80=>localhost:9000
accepted remote forwarding for tunnel 70f15fb59d2e7ebde55a6274ddfa54dd 80=>localhost:9000

After setting up the SSH Tunnel, I modified AbstractSeleniumTestCase's setUp() method to allow running tests on Sauce Labs.

@Before
public void setUp() throws Exception {
    if (SERVER == null) {
        SERVER = "localhost";
    }

    if (BROWSER == null) {
        BROWSER = "*firefox3";
    } else if (BROWSER.split(":").length == 3) {
        String[] platform = BROWSER.split(":");

        String os = platform[0];
        String browser = platform[1];

        // if Google Chrome, don't use a version #
        String version = (platform[1].equals("googlechrome") ? "" : platform[2]);
        String printableVersion = ((version.length() > 0) ? " " + platform[2].charAt(0) : "");

        String jobName = description.getMethodName() + " [" + browser + printableVersion + "]";

        BROWSER = "{\"username\":\"{your-username}\",\"access-key\":\"{your-access-key}\"," +
                "\"os\":\"" + platform[0] + "\",\"browser\": \"" + platform[1] + "\"," +
                "\"browser-version\":\"" + version + "\"," +
                "\"job-name\":\"" + jobName + "\"}";

        log.debug("Testing with " + browser + printableVersion + " on " + os);
    }

    if (APP == null) {
        APP = "http://localhost:9000";
    }

    selenium = new DefaultSelenium(SERVER, 4444, BROWSER, APP);
    selenium.start("captureNetworkTraffic=true");
    selenium.getEval("window.moveTo(1,1); window.resizeTo(1021,737);");
    selenium.setTimeout("60000");
}

After making this change, I was able to run Selenium tests from IDEA using the following steps:

1. Start Jetty on port 9000 (since that's what the tunnel points to). In IDEA's Maven panel, create a run/debug configuration for jetty:run, click the "Runner" tab and enter "-Djetty.port=9000" in the VM Parameters box.
2. Right-click on the test to run and create a run/debug configuration. Enter the following in the VM Parameters box. The last two parameters allow skipping the xvfb and Selenium RC startup process.
   -Dselenium.browser="Windows 2003:iexplore:8." -Dselenium.application=mraible.local -Dselenium.server=saucelabs.com -Dxvfb.skip=true -Dselenium.server.skip=true

These same parameters can be used if you want to run all tests from the command line:

mvn install -Pitest -Dselenium.browser="Windows 2003:iexplore:8." -Dselenium.application=mraible.local -Dselenium.server=saucelabs.com -Dxvfb.skip=true -Dselenium.server.skip=true -Dcargo.port=9000

To simplify things, we create profiles for the various browsers. For example, below are profiles for IE8 and Firefox 3.6.

<profile>
    <id>firefox-win</id>
    <properties>
        <cargo.port>9000</cargo.port>
        <selenium.application>http://${user.name}.local</selenium.application>
        <selenium.browser>Windows 2003:firefox:3.6.</selenium.browser>
        <selenium.server>saucelabs.com</selenium.server>
        <selenium.server.skip>true</selenium.server.skip>
        <xvfb.skip>true</xvfb.skip>
    </properties>
</profile>
<profile>
    <id>ie-win</id>
    <properties>
        <cargo.port>9000</cargo.port>
        <selenium.application>http://${user.name}.local</selenium.application>
        <selenium.browser>Windows 2003:iexplore:8.</selenium.browser>
        <selenium.server>saucelabs.com</selenium.server>
        <selenium.server.skip>true</selenium.server.skip>
        <xvfb.skip>true</xvfb.skip>
    </properties>
</profile>

Issues
Since we've started using Sauce Labs, we've run into a number of issues. Some of these are Selenium-related and some are simply things we learned since we started testing on multiple browsers.

• SSH Tunnels Keep Restarting This happens on our Hudson server that runs the tunnels as a service. This seems to happen daily and screws up our Hudson results because builds fail.
• XPath vs. CSS Selectors One of the first things we noticed was that our IE tests were 2-3 times slower than the same tests on Firefox. We discovered this is because Internet Explorer has a very slow XPath engine. To fix this issue, it's recommended that ids or CSS Selectors be used whenever trying to locate elements. For more information on CSS Selectors and Selenium, see CSS Selectors in Selenium Demystified. To test CSS Selectors, I found Firefinder to be a very useful Firefox plugin. Note that many pseudo elements won't work in IE.
• IE7 fails to initialize on Sauce Labs There's no errors in our JUnit reports, so we're not sure what's causing this. It could very well be bugs in our code/configuration, but IE8 works fine.
• The Job Names on Sauce Labs don't get set correctly and often results in duplicate job names. This could certainly be related to my code. Finding videos that show failed tests is difficult when the job names aren't set correctly.
• It would be slick if you could download the video of a failed test, similar to what we do by taking screenshots.
• Google Chrome works on Sauce Labs, but I'm unable to get it working locally (on Windows or OS X). This seems to be a Selenium issue.
• Safari 4 works, but when it fails, the screenshot shows a Safari can't find the file error. Since there's no real error to debug, it's difficult to figure out why the test fails. Since Safari 4 is not listed on platforms supported by Selenium, I'm unsure how to fix this.

Overall, Sauce Labs seems to work pretty well. However, in the process of messing with Hudson, build agents and Selenium infrastructure, it's become readily apparent that we need a team member to devote their full-attention to it. Having a developer or two work on it every now-and-then is inefficient, especially when we're still in the process of ironing everything out and making it all stable.

If you have any tips on how you've solved issues with Sauce Labs (ssh tunnels, IE7) or Selenium (Safari 4, Google Chrome), I'd love to hear them. I'm also interested to hear from anyone with experience running Selenium tests concurrently (locally or in the cloud).

Update: I discovered a bug in my AbstractSeleniumTest's setUp() method where job names weren't being set correctly. I've since changed the code in this class to the following:

private static String browser, printableVersion;

@BeforeClass
public static void parseBrowser() {

    if (BROWSER == null) {
        BROWSER = "*firefox3";
    } else if (BROWSER.split(":").length == 3) {
        String[] platform = BROWSER.split(":");

        String os = platform[0];
        browser = platform[1];

        // if Google Chrome, don't use a version #
        String version = (platform[1].equals("googlechrome") ? "" : platform[2]);
        printableVersion = ((version.length() > 0) ? " " + platform[2].charAt(0) : "");

        BROWSER = "{\"username\":\"{your-username}\",\"access-key\":\"{your-access-key}\"," +
                "\"os\":\"" + os + "\",\"browser\": \"" + browser + "\"," +
                "\"browser-version\":\"" + version + "\", " +
                "\"job-name\": \"jobName\"}";
    }
}

@Before
public void setUp() throws Exception {
    if (SERVER == null) {
        SERVER = "localhost";
    }

    if (APP == null) {
        APP = "http://localhost:9000";
    }

    String seleniumBrowser = BROWSER;
    if (BROWSER.startsWith("{")) { // sauce labs
        String jobName = description.getMethodName() + " [" + browser + printableVersion + "]";
        log.debug("=> Running job: " + jobName);

        seleniumBrowser = BROWSER.replace("jobName", jobName);
    }

    selenium = new DefaultSelenium(SERVER, 4444, seleniumBrowser, APP);
    selenium.start("captureNetworkTraffic=true");
    selenium.getEval("window.moveTo(1,1); window.resizeTo(1021,737);");
    selenium.setTimeout("60000");
}