Raible Designs

Developing Services with Apache Camel - Part III: Integrating Spring 4 and Spring Boot

2014-10-08T07:13:18-06:00

This article is the third in a series on Apache Camel and how I used it to replace IBM Message Broker for a client. I used Apache Camel for several months this summer to create a number of SOAP services. These services performed various third-party data lookups for our customers. For previous articles, see Part I: The Inspiration and Part II: Creating and Testing Routes.

In late June, I sent an email to my client's engineering team. Its subject: "External Configuration and Microservices". I recommended we integrate Spring Boot into the Apache Camel project I was working on. I told them my main motivation was its external configuration feature. I also pointed out its container-less WAR feature, where Tomcat (or Jetty) is embedded in the WAR and you can start your app with "java -jar appname.war". I mentioned microservices and that Spring Boot would make it easy to split the project into a project-per-service structure if we wanted to go that route. I then asked two simple questions:

Is it OK to integrate Spring Boot?
Should I split the project into microservices?

Both of these suggestions were well received, so I went to work.

Spring 4

Before I integrated Spring Boot, I knew I had to upgrade to Spring 4. The version of Camel I was using (2.13.1) did not support Spring 4. I found issue CAMEL-7074 (Support spring 4.x) and added a comment to see when it would be fixed. After fiddling with dependencies and trying Camel 2.14-SNAPSHOT, I was able to upgrade to CXF 3.0. However, this didn't solve my problem. There were some API uncompatible changes between Spring 3.3.x and Spring 4.0.x and the camel-test-spring module wouldn't work with both. I proposed the following:

I think the easiest way forward is to create two modules: camel-test-spring and camel-test-spring3. The former compiles against Spring 4 and the latter against Spring 3. You could switch it so camel-test-spring defaults to Spring 3, but camel-test-spring4 doesn't seem to be forward-looking, as you hopefully won't need a camel-test-spring5.

I've made this change in a fork and it works in my project. I can upgrade to Camel 2.14-SNAPSHOT and CXF 3.0 with Spring 3.2.8 (by using camel-test-spring3). I can also upgrade to Spring 4 if I use the upgraded camel-test-spring.

Here's a pull request that has this change: https://github.com/apache/camel/pull/199

The Camel team integrated my suggested change a couple weeks later. Unfortunately, a similar situation happened with Spring 4.1, so you'll have to wait for Camel 2.15 if you want to use Spring 4.1.

After making a patched 2.14-SNAPSHOT version available to my project, I was able to upgrade to Spring 4 and CXF 3 with a few minor changes to my pom.xml.

    <properties>
      <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
      <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
-     <camel.version>2.13.1</camel.version>
-     <cxf.version>2.7.11</cxf.version>
-     <spring.version>3.2.8.RELEASE</spring.version>
+     <camel.version>2.14-SNAPSHOT</camel.version>
+     <cxf.version>3.0.0</cxf.version>
+     <spring.version>4.0.5.RELEASE</spring.version>
    </properties>
...
+      <!-- upgrade camel-spring dependencies -->
+      <dependency>
+         <groupId>org.springframework</groupId>
+         <artifactId>spring-context</artifactId>
+         <version>${spring.version}</version>
+      </dependency>
+      <dependency>
+         <groupId>org.springframework</groupId>
+         <artifactId>spring-aop</artifactId>
+         <version>${spring.version}</version>
+      </dependency>
+      <dependency>
+         <groupId>org.springframework</groupId>
+         <artifactId>spring-tx</artifactId>
+         <version>${spring.version}</version>
+      </dependency>

I also had to change some imports for CXF 3.0 since it includes a new major version of Apache WSS4J (2.0.0).

-import org.apache.ws.security.handler.WSHandlerConstants;
+import org.apache.wss4j.dom.handler.WSHandlerConstants;
...
-import org.apache.ws.security.WSPasswordCallback;
+import org.apache.wss4j.common.ext.WSPasswordCallback;

After getting everything upgraded, I continued developing services for the next couple weeks.

Spring Boot

In late July, I integrated Spring Boot. It was fairly straightforward and mostly consisted of adding/removing dependencies and removing versions already defined in Boot's starter-parent.

+   <parent>
+      <groupId>org.springframework.boot</groupId>
+      <artifactId>spring-boot-starter-parent</artifactId>
+      <version>1.1.4.RELEASE</version>
+   </parent>
...
       <cxf.version>3.0.1</cxf.version>
+      <java.version>1.7</java.version>
+      <servlet-api.version>3.1.0</servlet-api.version>
       <spring.version>4.0.6.RELEASE</spring.version>
...
-            <artifactId>maven-compiler-plugin</artifactId>
-            <version>2.5.1</version>
-            <configuration>
-               <source>1.7</source>
-               <target>1.7</target>
-            </configuration>
-         </plugin>
-         <plugin>
-            <groupId>org.apache.maven.plugins</groupId>
             <artifactId>maven-resources-plugin</artifactId>
          </plugin>
+         <plugin>
+            <groupId>org.springframework.boot</groupId>
+            <artifactId>spring-boot-maven-plugin</artifactId>
+         </plugin>
       </plugins>
    </build>

    <dependencies>
+      <!-- spring boot -->
+      <dependency>
+         <groupId>org.springframework.boot</groupId>
+         <artifactId>spring-boot-starter-actuator</artifactId>
+         <exclusions>
+            <exclusion>
+               <groupId>org.springframework.boot</groupId>
+               <artifactId>spring-boot-starter-logging</artifactId>
+            </exclusion>
+         </exclusions>
+      </dependency>
+      <dependency>
+         <groupId>org.springframework.boot</groupId>
+         <artifactId>spring-boot-starter-log4j</artifactId>
+      </dependency>
+      <dependency>
+         <groupId>org.springframework.boot</groupId>
+         <artifactId>spring-boot-starter-tomcat</artifactId>
+         <scope>provided</scope>
+      </dependency>
+      <dependency>
+         <groupId>org.springframework.boot</groupId>
+         <artifactId>spring-boot-starter-web</artifactId>
+      </dependency>
       <!-- camel -->
...
-      <!-- upgrade camel-spring dependencies -->
-      <dependency>
-         <groupId>org.springframework</groupId>
-         <artifactId>spring-context</artifactId>
-         <version>${spring.version}</version>
-      </dependency>
-      <dependency>
-         <groupId>org.springframework</groupId>
-         <artifactId>spring-aop</artifactId>
-         <version>${spring.version}</version>
-      </dependency>
-      <dependency>
-         <groupId>org.springframework</groupId>
-         <artifactId>spring-tx</artifactId>
-         <version>${spring.version}</version>
-      </dependency>
...
-      <!-- logging -->
-      <dependency>
-         <groupId>org.slf4j</groupId>
-         <artifactId>slf4j-api</artifactId>
-         <version>1.7.6</version>
-      </dependency>
-      <dependency>
-         <groupId>org.slf4j</groupId>
-         <artifactId>slf4j-log4j12</artifactId>
-         <version>1.7.6</version>
-      </dependency>
-      <dependency>
-         <groupId>log4j</groupId>
-         <artifactId>log4j</artifactId>
-         <version>1.2.17</version>
-      </dependency>
-
       <!-- utilities -->
       <dependency>
          <groupId>joda-time</groupId>
          <artifactId>joda-time</artifactId>
-         <version>2.3</version>
       </dependency>
       <dependency>
          <groupId>commons-dbcp</groupId>
          <artifactId>commons-dbcp</artifactId>
-         <version>1.4</version>
...
       <!-- testing -->
       <dependency>
+         <groupId>org.springframework.boot</groupId>
+         <artifactId>spring-boot-starter-test</artifactId>
+         <exclusions>
+            <exclusion>
+               <groupId>org.springframework.boot</groupId>
+               <artifactId>spring-boot-starter-logging</artifactId>
+            </exclusion>
+         </exclusions>
+      </dependency>
+      <dependency>
-      <dependency>
-         <groupId>org.springframework</groupId>
-         <artifactId>spring-test</artifactId>
-         <version>${spring.version}</version>
-         <scope>test</scope>
-      </dependency>
-      <dependency>
-         <groupId>org.mockito</groupId>
-         <artifactId>mockito-core</artifactId>
-         <version>1.9.5</version>
-         <scope>test</scope>
-      </dependency>

Next, I deleted the AppInitializer.java class I mentioned in Part II and added an Application.java class.

import org.apache.cxf.transport.servlet.CXFServlet;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration;
import org.springframework.boot.autoconfigure.jdbc.DataSourceTransactionManagerAutoConfiguration;
import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.boot.context.embedded.ConfigurableEmbeddedServletContainer;
import org.springframework.boot.context.embedded.EmbeddedServletContainerCustomizer;
import org.springframework.boot.context.embedded.ErrorPage;
import org.springframework.boot.context.embedded.ServletRegistrationBean;
import org.springframework.boot.context.web.SpringBootServletInitializer;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Configuration;
import org.springframework.http.HttpStatus;

@Configuration
@EnableAutoConfiguration(exclude = {DataSourceAutoConfiguration.class,
        DataSourceTransactionManagerAutoConfiguration.class})
@ComponentScan
public class Application extends SpringBootServletInitializer {

    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }

    @Override
    protected SpringApplicationBuilder configure(SpringApplicationBuilder application) {
        return application.sources(Application.class);
    }

    @Bean
    public ServletRegistrationBean servletRegistrationBean() {
        CXFServlet servlet = new CXFServlet();
        return new ServletRegistrationBean(servlet, "/api/*");
    }

    @Bean
    public EmbeddedServletContainerCustomizer containerCustomizer() {
        return new EmbeddedServletContainerCustomizer() {
            @Override
            public void customize(ConfigurableEmbeddedServletContainer container) {
                ErrorPage error401Page = new ErrorPage(HttpStatus.UNAUTHORIZED, "/401.html");
                ErrorPage error404Page = new ErrorPage(HttpStatus.NOT_FOUND, "/404.html");
                ErrorPage error500Page = new ErrorPage(HttpStatus.INTERNAL_SERVER_ERROR, "/500.html");
                container.addErrorPages(error401Page, error404Page, error500Page);
            }
        };
    }
}

The error pages you see configured above were configured and copied from Tim Sporcic's Custom Error Pages with Spring Boot.

Dynamic DataSources

I excluded the DataSource-related AutoConfiguration classes because this application had many datasources. It also had a requirement to allow datasources to be added on-the-fly by simply editing application.properties. I asked how to do this on Stack Overflow and received an excellent answer from Stéphane Nicoll.

Spring Boot Issues

I did encounter a couple issues after integrating Spring Boot. The first was that it was removing the content-* headers for CXF responses. This only happened when running the WAR in Tomcat and I was able to figure out a workaround with a custom ResponseWrapper and Filter. This issue was fixed in Spring Boot 1.1.6.

The other issue was that the property override feature didn't seem to work when setting environment variables. The workaround was to create a setenv.sh script in $CATALINA_HOME/bin and add the environment variables there. See section 3.4 of Tomcat 7's RUNNING.txt for more information.

SOAP Faults

After upgrading to Spring 4 and integrating Spring Boot, I continued migrating IBM Message Broker flows. My goal was to make all new services backward compatible, but I ran into an issue. With the new services, SOAP Faults were sent back to the client instead of error messages in a SOAP Message. I suggested we fix it with one of two ways:

Modify the client so it looks for SOAP Faults and handles them appropriately.
Modify the new services so messages are returned instead of faults.

For #2, I learned how do to convert from a fault to messages on the Camel user mailing list. However, the team opted to improve the client and we added fault handling there instead.

Microservice Deployment

When I first integrated Spring Boot, I was planning on splitting our project into a project-per-service. This would allow each service to evolve on its own, instead of having a monolithic war that contains all the services. In team discussions, there was some concern about the memory overhead of running multiple instances instead of one.

I pointed out an interesting thread on the Camel mailing list about deploying routes with a route-per-jvm or all in the same JVM. The recommendation from that thread was to bundle similar routes together if you were to split them.

In the end, we decided to allow our Operations team decide how they wanted to manage/deploy everything. I mentioned that Spring Boot can work with Tomcat, Jetty, JBoss and even cloud providers like Heroku and Cloud Foundry. I estimated that splitting the project apart would take less than a day, as would making it back into a monolithic WAR.

Summary

This article explains how we upgraded our Apache Camel application to Spring 4 and integrated Spring Boot. There was a bit of pain getting things to work, but nothing a few pull requests and workarounds couldn't fix. We discovered some issues with setting environment variables for Tomcat and opted not to split our project into small microservices. Hopefully this article will help people trying to Camelize a Spring Boot application .

In the next article, I'll talk about load testing with Gatling, logging with Log4j2 and monitoring with hawtio and New Relic.

Developing Services with Apache Camel - Part II: Creating and Testing Routes

2014-09-30T10:05:38-06:00

This article is the second in a series on Apache Camel and how I used it to replace IBM Message Broker for a client. The first article, Developing Services with Apache Camel - Part I: The Inspiration, describes why I chose Camel for this project.

To make sure these new services correctly replaced existing services, a 3-step approach was used:

Write an integration test pointing to the old service.
Write the implementation and a unit test to prove it works.
Write an integration test pointing to the new service.

I chose to start by replacing the simplest service first. It was a SOAP Service that talked to a database to retrieve a value based on an input parameter. To learn more about Camel and how it works, I started by looking at the CXF Tomcat Example. I learned that Camel is used to provide routing of requests. Using its CXF component, it can easily produce SOAP web service endpoints. An end point is simply an interface, and Camel takes care of producing the implementation.

Legacy Integration Test

I started by writing a LegacyDrugServiceTests integration test for the old drug service. I tried two different ways of testing, using WSDL-generated Java classes, as well as using JAX-WS's SOAP API. Finding the WSDL for the legacy service was difficult because IBM Message Broker doesn't expose it when adding "?wsdl" to the service's URL. Instead, I had to dig through the project files until I found it. Then I used the cxf-codegen-plugin to generate the web service client. Below is what one of the tests looked like that uses the JAX-WS API.

@Test
public void sendGPIRequestUsingSoapApi() throws Exception {
    SOAPElement bodyChildOne = getBody(message).addChildElement("gpiRequest", "m");
    SOAPElement bodyChildTwo = bodyChildOne.addChildElement("args0", "m");
    bodyChildTwo.addChildElement("NDC", "ax22").addTextNode("54561237201");
    SOAPMessage reply = connection.call(message, getUrlWithTimeout(SERVICE_NAME));
    if (reply != null) {
        Iterator itr = reply.getSOAPBody().getChildElements();
        Map resultMap = TestUtils.getResults(itr);
        assertEquals("66100525123130", resultMap.get("GPI"));
    }
}

Implementing the Drug Service

In the last article, I mentioned I wanted no XML in the project. To facilitate this, I used Camel's Java DSL to define routes and Spring's JavaConfig to configure dependencies.

The first route I wrote was one that looked up a GPI (Generic Product Identifier) by NDC (National Drug Code).

@WebService
public interface DrugService {
    @WebMethod(operationName = "gpiRequest")
    GpiResponse findGpiByNdc(GpiRequest request);
}

To expose this as a web service endpoint with CXF, I needed to do two things:

Tell Spring how to configure CXF by importing "classpath:META-INF/cxf/cxf.xml" into a @Configuration class.
Configure CXF's Servlet so endpoints can be served up at a particular URL.

To satisfy item #1, I created a CamelConfig class that extends CamelConfiguration. This class allows Camel to be configured by Spring's JavaConfig. In it, I imported the CXF configuration, allowed tracing to be configured dynamically, and exposed my application.properties to Camel. I also set it up (with @ComponentScan) to look for Camel routes annotated with @Component.

@Configuration
@ImportResource("classpath:META-INF/cxf/cxf.xml")
@ComponentScan("com.raibledesigns.camel")
public class CamelConfig extends CamelConfiguration {
    @Value("${logging.trace.enabled}")
    private Boolean tracingEnabled;

    @Override
    protected void setupCamelContext(CamelContext camelContext) throws Exception {
        PropertiesComponent pc = new PropertiesComponent();
        pc.setLocation("classpath:application.properties");
        camelContext.addComponent("properties", pc);
        // see if trace logging is turned on
        if (tracingEnabled) {
            camelContext.setTracing(true);
        }
        super.setupCamelContext(camelContext);
    }

    @Bean
    public Tracer camelTracer() {
        Tracer tracer = new Tracer();
        tracer.setTraceExceptions(false);
        tracer.setTraceInterceptors(true);
        tracer.setLogName("com.raibledesigns.camel.trace");
        return tracer;
    }
}

CXF has a servlet that's responsible for serving up its services at common path. To map CXF's servlet, I leveraged Spring's WebApplicationInitializer in an AppInitializer class. I decided to serve up everything from a /api/* base URL.

package com.raibledesigns.camel.config;

import org.apache.cxf.transport.servlet.CXFServlet;
import org.springframework.web.WebApplicationInitializer;
import org.springframework.web.context.ContextLoaderListener;
import org.springframework.web.context.support.AnnotationConfigWebApplicationContext;

import javax.servlet.ServletContext;
import javax.servlet.ServletException;
import javax.servlet.ServletRegistration;

public class AppInitializer implements WebApplicationInitializer {

    @Override
    public void onStartup(ServletContext servletContext) throws ServletException {
        servletContext.addListener(new ContextLoaderListener(getContext()));
        ServletRegistration.Dynamic servlet = servletContext.addServlet("CXFServlet", new CXFServlet());
        servlet.setLoadOnStartup(1);
        servlet.setAsyncSupported(true);
        servlet.addMapping("/api/*");
    }

    private AnnotationConfigWebApplicationContext getContext() {
        AnnotationConfigWebApplicationContext context = new AnnotationConfigWebApplicationContext();
        context.setConfigLocation("com.raibledesigns.camel.config");
        return context;
    }
}

To implement this web service with Camel, I created a DrugRoute class that extends Camel's RouteBuilder.

@Component
public class DrugRoute extends RouteBuilder {
    private String uri = "cxf:/drugs?serviceClass=" + DrugService.class.getName();

    @Override
    public void configure() throws Exception {
        from(uri)
            .recipientList(simple("direct:${header.operationName}"));
        from("direct:gpiRequest").routeId("gpiRequest")
            .process(new Processor() {
                public void process(Exchange exchange) throws Exception {
                    // get the ndc from the input
                    String ndc = exchange.getIn().getBody(GpiRequest.class).getNDC();
                    exchange.getOut().setBody(ndc);
                }
            })
            .to("sql:{{sql.selectGpi}}")
            .to("log:output")
            .process(new Processor() {
                public void process(Exchange exchange) throws Exception {
                    // get the gpi from the input
                    List<HashMap> data = (ArrayList<HashMap>) exchange.getIn().getBody();
                    DrugInfo drug = new DrugInfo();
                    if (data.size() > 0) {
                        drug = new DrugInfo(String.valueOf(data.get(0).get("GPI")));
                    }
                    GpiResponse response = new GpiResponse(drug);
                    exchange.getOut().setBody(response);
                }
            });
    }
}

The sql.selectGpi property is read from src/main/resources/application.properties and looks as follows:

sql.selectGpi=select GPI from drugs where ndc = #?dataSource=ds.drugs

The "ds.drugs" reference is to a datasource that's created by Spring. From my AppConfig class:

@Configuration
@PropertySource("classpath:application.properties")
public class AppConfig {

    @Value("${ds.driver.db2}")
    private String jdbcDriverDb2;

    @Value("${ds.password}")
    private String jdbcPassword;

    @Value("${ds.url}")
    private String jdbcUrl;

    @Value("${ds.username}")
    private String jdbcUsername;

    @Bean(name = "ds.drugs")
    public DataSource drugsDataSource() {
        return createDataSource(jdbcDriverDb2, jdbcUsername, jdbcPassword, jdbcUrl);
    }

    private BasicDataSource createDataSource(String driver, String username, String password, String url) {
        BasicDataSource ds = new BasicDataSource();
        ds.setDriverClassName(driver);
        ds.setUsername(username);
        ds.setPassword(password);
        ds.setUrl(url);
        ds.setMaxActive(100);
        ds.setMaxWait(1000);
        ds.setPoolPreparedStatements(true);
        return ds;
    }
}

Unit Testing

The hardest part about unit testing this route was figuring out how to use Camel's testing support. I posted a question to the Camel users mailing list in early June. Based on advice received, I bought Camel in Action, read chapter 6 on testing and went to work. I wanted to eliminate the dependency on a datasource, so I used Camel's AdviceWith feature to modify my route and intercept the SQL call. This allowed me to return pre-defined results and verify everything worked.

@RunWith(CamelSpringJUnit4ClassRunner.class)
@ContextConfiguration(loader = CamelSpringDelegatingTestContextLoader.class, classes = CamelConfig.class)
@DirtiesContext(classMode = DirtiesContext.ClassMode.AFTER_EACH_TEST_METHOD)
@UseAdviceWith
public class DrugRouteTests {

    @Autowired
    CamelContext camelContext;

    @Produce
    ProducerTemplate template;

    @EndpointInject(uri = "mock:result")
    MockEndpoint result;

    static List<Map> results = new ArrayList<Map>() {{
        add(new HashMap<String, String>() {{
            put("GPI", "123456789");
        }});
    }};

    @Before
    public void before() throws Exception {
        camelContext.setTracing(true);

        ModelCamelContext context = (ModelCamelContext) camelContext;
        RouteDefinition route = context.getRouteDefinition("gpiRequest");
        route.adviceWith(context, new RouteBuilder() {
            @Override
            public void configure() throws Exception {
                interceptSendToEndpoint("sql:*").skipSendToOriginalEndpoint().process(new Processor() {
                    @Override
                    public void process(Exchange exchange) throws Exception {
                        exchange.getOut().setBody(results);
                    }
                });
            }
        });
        route.to(result);
        camelContext.start();
    }

    @Test
    public void testMockSQLEndpoint() throws Exception {
        result.expectedMessageCount(1);
        GpiResponse expectedResult = new GpiResponse(new DrugInfo("123456789"));
        result.allMessages().body().contains(expectedResult);

        GpiRequest request = new GpiRequest();
        request.setNDC("123");
        template.sendBody("direct:gpiRequest", request);

        MockEndpoint.assertIsSatisfied(camelContext);
    }
}

I found AdviceWith to be extremely useful as I developed more routes and tests in this project. I used its weaveById feature to intercept calls to stored procedures, replace steps in my routes and remove steps I didn't want to test. For example, in one route, there was a complicated workflow to interact with a customer's data.

Call a stored procedure in a remote database, which then inserts a record into a temp table.
Lookup that data using the value returned from the stored procedure.
Delete the record from the temp table.
Parse the data (as CSV) since the returned value is ~ delimited.
Convert the parsed data into objects, then do database inserts in a local database (if data doesn't exist).

To make matters worse, remote database access was restricted by IP address. This meant that, while developing, I couldn't even manually test from my local machine. To solve this, I used the following:

interceptSendToEndpoint("bean:*") to intercept the call to my stored procedure bean.
weaveById("myJdbcProcessor").before() to replace the temp table lookup with a CSV file.
Mockito to mock a JdbcTemplate that does the inserts.

To figure out how to configure and execute stored procedures in a route, I used the camel-store-procedure project on GitHub. Mockito's ArgumentCaptor also became very useful when developing a route that called a 3rd-party web service within a route. James Carr has more information on how you might use this to verify values on an argument.

To see if my tests were hitting all aspects of the code, I integrated the cobertura-maven-plugin for code coverage reports (generated by running mvn site).

<build>
    <plugins>
        <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>cobertura-maven-plugin</artifactId>
            <configuration>
                <instrumentation>
                    <excludes>
                        <exclude>**/model/*.class</exclude>
                        <exclude>**/AppInitializer.class</exclude>
                        <exclude>**/StoredProcedureBean.class</exclude>
                        <exclude>**/SoapActionInterceptor.class</exclude>
                    </excludes>
                </instrumentation>
                <check/>
            </configuration>
            <version>2.6</version>
        </plugin>
...
<reporting>
    <plugins>
        <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>cobertura-maven-plugin</artifactId>
            <version>2.6</version>
        </plugin>

Integration Testing

Writing an integration test was fairly straightforward. I created a DrugRouteITest class, a client using CXF's JaxWsProxyFactoryBean and called the method on the service.

public class DrugRouteITest {

    private static final String URL = "http://localhost:8080/api/drugs";

    protected static DrugService createCXFClient() {
        JaxWsProxyFactoryBean factory = new JaxWsProxyFactoryBean();
        factory.setBindingId("http://schemas.xmlsoap.org/wsdl/soap12/");
        factory.setServiceClass(DrugService.class);
        factory.setAddress(getTestUrl(URL));
        return (DrugService) factory.create();
    }

    @Test
    public void findGpiByNdc() throws Exception {
        // create input parameter
        GpiRequest input = new GpiRequest();
        input.setNDC("54561237201");

        // create the webservice client and send the request
        DrugService client = createCXFClient();
        GpiResponse response = client.findGpiByNdc(input);

        assertEquals("66100525123130", response.getDrugInfo().getGPI());
    }
}

This integration test is only run after Tomcat has started and deployed the app. Unit tests are run by Maven's surefire-plugin, while integration tests are run by the failsafe-plugin. An available Tomcat port is determined by the build-helper-maven-plugin. This port is set as a system property and read by the getTestUrl() method call above.

public static String getTestUrl(String url) {
    if (System.getProperty("tomcat.http.port") != null) {
        url = url.replace("8080", System.getProperty("tomcat.http.port"));
    }
    return url;
}

Below are the relevant bits from pom.xml that determines when to start/stop Tomcat, as well as which tests to run.

<plugin>
    <groupId>org.apache.tomcat.maven</groupId>
    <artifactId>tomcat7-maven-plugin</artifactId>
    <version>2.2</version>
    <configuration>
        <path>/</path>
    </configuration>
    <executions>
        <execution>
            <id>start-tomcat</id>
            <phase>pre-integration-test</phase>
            <goals>
                <goal>run</goal>
            </goals>
            <configuration>
                <fork>true</fork>
                <port>${tomcat.http.port}</port>
            </configuration>
        </execution>
        <execution>
            <id>stop-tomcat</id>
            <phase>post-integration-test</phase>
            <goals>
                <goal>shutdown</goal>
            </goals>
        </execution>
    </executions>
</plugin>
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.17</version>
    <configuration>
        <excludes>
            <exclude>**/*IT*.java</exclude>
            <exclude>**/Legacy**.java</exclude>
        </excludes>
        <includes>
            <include>**/*Tests.java</include>
            <include>**/*Test.java</include>
        </includes>
    </configuration>
</plugin>
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-failsafe-plugin</artifactId>
    <version>2.17</version>
    <configuration>
        <includes>
            <include>**/*IT*.java</include>
        </includes>
        <systemProperties>
            <tomcat.http.port>${tomcat.http.port}</tomcat.http.port>
        </systemProperties>
    </configuration>
    <executions>
        <execution>
            <goals>
                <goal>integration-test</goal>
                <goal>verify</goal>
            </goals>
        </execution>
    </executions>
</plugin>

The most useful part of integration testing came when I copied one of my legacy tests into it and started verifying backwards compatibility. Since we wanted to replace existing services, and require no client changes, I had to make the XML request and response match. Charles was very useful for this exercise, letting me inspect the request/response and tweak things to match. The following JAX-WS annotations allowed me to change the XML element names and achieve backward compatibility.

@BindingType(SOAPBinding.SOAP12HTTP_BINDING)
@WebResult(name = "return", targetNamespace = "...")
@ResponseWrapper(localName = "gpiResponse")
@WebParam(name = "args0", targetNamespace = "...")
@XmlElement(name = "...")

Continuous Integration and Deployment

My next item of business was configuring a job in Jenkins to continually test and deploy. Getting all the tests to pass was easy, and deploying to Tomcat was simple enough thanks to the Deploy Plugin and this article. However, after a few deploys, Tomcat would throw OutOfMemory exceptions. Therefore, I ended up creating a second "deploy" job that stops Tomcat, copies the successfully-built WAR to $CATALINA_HOME/webapps, removes $CATALINA_HOME/webapps/ROOT and restarts Tomcat. I used Jenkins "Execute shell" feature to configure these three steps. I was pleased to find my /etc/init.d/tomcat script still worked for starting Tomcat at boot time and providing convenient start/stop commands.

Summary

This article shows you how I implemented and tested a simple Apache Camel route. The route described only does a simple database lookup, but you can see how Camel's testing support allows you to mock results and concentrate on developing your route logic. I found its testing framework very useful and not well documented, so hopefully this article helps to fix that. In the next article, I'll talk about upgrading to Spring 4, integrating Spring Boot and our team's microservice deployment discussions.

Fixing XSS in JSP 2

2011-02-28T14:08:46-07:00

Way back in 2007, I wrote about Java Web Frameworks and XSS. My main point was that JSP EL doesn't bother to handle XSS.

Of course, the whole problem with JSP EL could be solved if Tomcat (and other containers) would allow a flag to turn on XML escaping by default. IMO, it's badly needed to make JSP-based webapps safe from XSS.

A couple months later, I proposed a Tomcat enhancement to escape JSP's EL by default. I also entered an enhancement request for this feature and attached a patch. That issue has remained open and unfixed for 3 and 1/2 years.

Yesterday, Chin Huang posted a handy-dandy ELResolver that XML-escapes EL values.

I tried Chin's resolver in AppFuse today and it works as well as advertised. To do this, I copied his EscapeXML*.java files into my project, changed the JSP API's Maven coordinates from javax.servlet:jsp-api:2.0 to javax.servlet.jsp:jsp-api:2.1 and added the listener to web.xml.

With Struts 2 and Spring MVC, I was previously able to have ${param.xss} and pass in ?xss=<script>alert('gotcha')</script> and it would show a JavaScript alert. After using Chin's ELResolver, it prints the string on the page instead of displaying an alert.

Thanks to Chin Huang for this patch! If you're using JSP, I highly recommend you add this to your projects as well.

Integration Testing with HTTP, HTTPS and Maven

2011-02-11T15:54:16-07:00

Earlier this week, I was tasked with getting automated integration tests working in my project at Overstock.com. By automated, I mean that ability to run "mvn install" and have the following process cycled through:

Start a container
Deploy the application
Run all integration tests
Stop the container

Since it makes sense for integration tests to run in Maven's integration-test phase, I first configured the maven-surefire-plugin to skip tests in the test phase and execute them in the integration-test phase. I used the <id>default-phase</id> syntax to override the plugins' usual behavior.

<plugin>
  <artifactId>maven-surefire-plugin</artifactId>
  <executions>
    <execution>
      <id>default-test</id>
      <configuration>
        <excludes>
          <exclude>**/*Test*.java</exclude>
        </excludes>
      </configuration>
    </execution>
    <execution>
      <id>default-integration-test</id>
      <phase>integration-test</phase>
      <goals>
        <goal>test</goal>
      </goals>
      <configuration>
        <includes>
          <include>**/*Test.java</include>
        </includes>
        <excludes>
          <exclude>none</exclude>
          <exclude>**/TestCase.java</exclude>
        </excludes>
      </configuration>
    </execution>
  </executions>
</plugin>

After I had this working, I moved onto getting the container started and stopped properly. In the past, I've done this using Cargo and it's always worked well for me. Apart from the usual setup I use in AppFuse archetypes (example pom.xml), I added a couple additional items:

Added <timeout>180000</timeout> so the container would wait up to 3 minutes for the WAR to deploy.
In configuration/properties, specified <context.path>ROOT</context.path> so the app would deploy at the / context path.
In configuration/properties, specified <cargo.protocol>https</cargo.protocol> since many existing unit tests made requests to secure resources.

I started by using Cargo with Tomcat and had to create certificate keystore in order to get Tomcat to start with SSL enabled. After getting it to start, I found the tests failed with the following errors in the logs:

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: 
PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: 
unable to find valid certification path to requested target
	at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1649)

Co-workers told me this was easily solved by adding my 'untrusted' cert to my JVM keystore. Once all this was working, I thought I was good to go, but found that some tests were still failing. The failures turned out to be because they were talking to http and https was the only protocol enabled. After doing some research, I discovered that Cargo doesn't support starting on both http and https ports.

So back to the drawing board I went. I ended up turning to the maven-jetty-plugin and the tomcat-maven-plugin to get the functionality I was looking for. I also automated the certificate keystore generation using the keytool-maven-plugin. Below is the extremely-verbose 95-line profiles section of my pom.xml that allows either container to be used.

Sidenote: I wonder how this same setup would look using Gradle?

<profiles>
  <profile>
    <id>jetty</id>
    <activation>
      <activeByDefault>true</activeByDefault>
    </activation>
    <build>
      <plugins>
        <plugin>
          <groupId>org.mortbay.jetty</groupId>
          <artifactId>maven-jetty-plugin</artifactId>
          <version>6.1.26</version>
          <configuration>
            <contextPath>/</contextPath>
            <connectors>
              <connector implementation="org.mortbay.jetty.nio.SelectChannelConnector">
                <!-- forwarded == true interprets x-forwarded-* headers -->
                <!-- http://docs.codehaus.org/display/JETTY/Configuring+mod_proxy -->
                <forwarded>true</forwarded>
                <port>8080</port>
                <maxIdleTime>60000</maxIdleTime>
              </connector>
              <connector implementation="org.mortbay.jetty.security.SslSocketConnector">
                <forwarded>true</forwarded>
                <port>8443</port>
                <maxIdleTime>60000</maxIdleTime>
                <keystore>${project.build.directory}/ssl.keystore</keystore>
                <password>overstock</password>
                <keyPassword>overstock</keyPassword>
              </connector>
            </connectors>
            <stopKey>overstock</stopKey>
            <stopPort>9999</stopPort>
          </configuration>
          <executions>
            <execution>
              <id>start-jetty</id>
              <phase>pre-integration-test</phase>
              <goals>
                <goal>run-war</goal>
              </goals>
              <configuration>
                <daemon>true</daemon>
              </configuration>
            </execution>
            <execution>
              <id>stop-jetty</id>
              <phase>post-integration-test</phase>
              <goals>
                <goal>stop</goal>
              </goals>
            </execution>
          </executions>
        </plugin>
      </plugins>
    </build>
  </profile>
  <profile>
    <id>tomcat</id>
    <build>
      <plugins>
        <plugin>
          <groupId>org.codehaus.mojo</groupId>
          <artifactId>tomcat-maven-plugin</artifactId>
          <version>1.1</version>
          <configuration>
            <addContextWarDependencies>true</addContextWarDependencies>
            <fork>true</fork>
            <path>/</path>
            <port>8080</port>
            <httpsPort>8443</httpsPort>
            <keystoreFile>${project.build.directory}/ssl.keystore</keystoreFile>
            <keystorePass>overstock</keystorePass>
          </configuration>
          <executions>
            <execution>
              <id>start-tomcat</id>
              <phase>pre-integration-test</phase>
              <goals>
                <goal>run-war</goal>
              </goals>
            </execution>
            <execution>
              <id>stop-tomcat</id>
              <phase>post-integration-test</phase>
              <goals>
                <goal>shutdown</goal>
              </goals>
            </execution>
          </executions>
        </plugin>
      </plugins>
    </build>
  </profile>
</profiles>

With this setup in place, I was able to automate running our integration tests by simply typing "mvn install" (for Jetty) or "mvn install -Ptomcat" (for Tomcat). For running in Hudson, it's possible I'll have to further enhance things to randomize the port and pass that into tests as a system property. The build-helper-maven-plugin and its reserve-network-port goal is a nice way to do this. Note: if you want to run more than one instance of Tomcat at a time, you might have to randomize the ajp and rmi ports to avoid collisions.

The final thing I encountered was our app didn't shutdown gracefully. Luckily, this was fixed in a newer version of our core framework and upgrading fixed the problem. Here's the explanation from an architect on the core framework team.

The hanging problem was caused by the way the framework internally aggregated statistics related to database connection usage and page response times. The aggregation runs on a separate thread but not as a daemon thread. Previously, the aggregation threads weren't being terminated on shutdown so the JVM would hang waiting for them to finish. In the new frameworks, the aggregation threads are terminated on shutdown.

Hopefully this post helps you test your secure and unsecure applications at the same time. At the same time, I'm hoping it motivates the Cargo developers to add simultaneous http and https support.

Update: In the comments, Ron Piterman recommended I use the Maven Failsafe Plugin because its designed to run integration tests while Surefire Plugin is for unit tests. I changed my configuration to the following and everything still passes. Thanks Ron!

<plugin>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>2.7.2</version>
  <configuration>
    <skipTests>true</skipTests>
  </configuration>
</plugin>
<plugin>
  <artifactId>maven-failsafe-plugin</artifactId>
  <version>2.7.2</version>
  <configuration>
    <includes>
      <include>**/*Test.java</include>
    </includes>
    <excludes>
      <exclude>**/TestCase.java</exclude>
    </excludes>
  </configuration>
  <executions>
    <execution>
      <id>integration-test</id>
      <phase>integration-test</phase>
      <goals>
        <goal>integration-test</goal>
      </goals>
    </execution>
    <execution>
      <id>verify</id>
      <phase>verify</phase>
      <goals>
        <goal>verify</goal>
      </goals>
    </execution>
  </executions>
</plugin>

Update 2: In addition to application changes to solve hanging issues, I also had to change my Jetty Plugin configuration to use a different SSL connector implementation. This also required adding the jetty-sslengine dependency, which has been renamed to jetty-ssl for Jetty 7.

<connector implementation="org.mortbay.jetty.security.SslSelectChannelConnector">
...
<dependencies>
  <dependency>
    <groupId>org.mortbay.jetty</groupId>
    <artifactId>jetty-sslengine</artifactId>
    <version>6.1.26</version>
  </dependency>
</dependencies>

What's New in Spring 3.0

2010-03-19T11:46:25-06:00

This morning, I attended Rod Johnson's What's New in Spring 3.0 keynote at TSSJS. Rod ditched his slides for the talk and mentioned that this might be risky. Especially since he was pretty jetlagged (flew in from Paris at 11pm last night). Below are my notes from his talk.

The most important thing for the future of Java is productivity and cloud computing. The focus at SpringSource is heavily on productivity and not just on improving the Spring codebase. If you look at the comparisons out there between Rails and Spring, it's not an apples-to-apples comparison. The philosophy with Spring has always been the developer is always right. However, if you look at something like Rails, you'll see it's far more prescriptive. That layer of opinionated frameworks is important in that it improves your productivity greatly.

SpringSource is putting a lot of emphasis on improving developer productivity with two opinionated frameworks: Grails and Spring Roo. To show how productive developers can be, Rod started to build a web app with Spring Roo. As part of this demo, he mentioned we'd see many of the new features of Spring 3: RestTemplate, @Value and Spring EL.

Rod used STS to write the application and built a Twitter client. After creating a new project using File -> New Roo Project, a Roo Shell tab shows up at the bottom. Typing "hint" tells you what you should do write away. The initial message is "Roo requires the installation of a JPA provider and associated database." The initial command is "persistence setup --provider HIBERNATE --database HYPERSONIC_IN_MEMORY". After running this, a bunch of log messages are shown on the console, most of them indicating that pom.xml has been modified.

The first file that Rod shows is src/main/resources/META-INF/spring/applicationContext.xml. It's the only XML file you'll need in your application and includes a PropertyPlaceHolderConfigurer, a context:component-scan for finding annotations and a transaction manager.

After typing "hint" again, Roo indicates that Rod should create entities. He does this by running "ent --class ~.domain.Term --testAutomatically". A Term class (with a bunch of annotations) is created, as well as a number of *.aj files and an integration test. Most of the files don't have anything in them but annotations. The integration test uses @RooIntegrationTest(entity=Term.class) on its class to fire up a Spring container in the test and do dependency injection (if necessary). From there, Rod demonstrated that he could easily modify the test to verify the database existed.

private SimpleJdbcTemplate jt;

@Autowired
public void init(DataSource ds) {
    this.jt = new SimpleJdbcTemplate(ds);
}

@Test 
public void testDb() {
    jt.queryForInt("SELECT COUNT(0) FROM TERM");
}

Interestingly, after running the test, you could see a whole bunch of tests being run, not just the one that was in the class itself. From there, he modified the Term class to add two new properties: name and searchTerms. He also used JSR 303's @NotNull annotation to make the fields required.

@Entity
@RooJavaBean
@RooToString
@RooEntity
public class Term {

    @NotNull
    private String name;

    @NotNull
    private String searchTerms;
}

Next, Rod added a new test and showed that the setters for these properties were automatically created and he never had to write getters and setters. This is done by aspects that are generated beside your Java files. Roo is smart enough that if you write toString() methods in your Java code, it will delete the aspect that normally generates the toString() method.

To add fields to an entity from the command lie, you can run commands like "field string --fieldName text --notNull" and "field number --type java.lang.Long --fieldName twitterId --notNull". The Roo Shell is also capable of establishing relationships between entities.

After successfully modifying his Entities, Rod started creating code to talk to Twitter's API. He used RestTemplate to do this and spent a good 5 minutes trying to get Eclipse to import the class properly. The best part of this demo was watching him do what most developers do: searching Google for RestTemplate to get the package name to import.

After awkward silence and some fumbling, he opened an existing project (that had the dependencies properly configured) and used Java Config to configure beans for the project. This was done with a @Configuration annotation on the class, @Value annotations on properties (that read from a properties file) and @Bean annotations for the beans to expose. The first time Rod tried to run the test it failed because a twitter.properties file didn't exist. After creating it, he successfully ran the test and successfully searched Twitter's API.

The nice thing about @Configuration is the classes are automatically picked up and you don't need to configure any XML to recognize them. Also, in your Java classes, you don't have to use @Autowired to get @Bean references injected.

After this, Rod attempted to show a web interface of the application. He started the built-in SpringSource tc Server and proceeded to show us Tomcat's 404 page. Unfortunately, Tomcat seemed to startup OK (no errors in the logs), but obviously something didn't work well. For the next few silent moments, we watched him try to delete web.xml from Eclipse. Unfortunately, this didn't work and we weren't able to see the scaffolding the entities that Rod created.

At this point, Rod opened a completed version of the app and was able to show it to us in a browser. You could hear the murmur of the crowd as everyone realized he was about to show the the Twitter search results for #tssjs. Most of the tweets displayed were from folks commenting about how some things didn't work in the demo.

In summary, there's some really cool things in Spring 3: @Configuration, @Value, task scheduling with @Scheduled and one-way methods with @Async.

Final points of SpringSource and VMWare: they're committed to Java and middleware. Their big focus is providing an integrated experience from productivity to cloud. There's other languages that are further along than Java and SpringSource is trying to fix that. One thing they're working on is a private Java cloud that companies can use and leverage as a VMWare appliance.

I think there's a lot of great things in Spring 3 and most users of Roo seem to be happy with it. It's unfortunate that the Demo Gods frowned upon Rod, but it was cool to see him do the "no presentation" approach.

Packaging a SOFEA Application for Distribution

2009-10-06T01:17:38-06:00

The project I'm working on is a bit different from those I'm used to. I'm used to working on web applications that are hosted on servers and customers access with their browser. SaaS if you will. My current client is different. They're a product company that sells applications and distributes them to customers via download and CD. Their customers install these applications on internal servers (supported servers include WebSphere, WebLogic and Tomcat).

The product I'm currently working on is structured as a SOFEA application and therefore consists of two separate modules - a backend and a frontend. Since it's installed in a servlet container, both modules are WARs and can be installed separately.

Building the backend and frontend as separate projects makes a lot of sense for two reasons:

In development, different teams can work on the frontend and backend projects.
Having them as separate projects allows them to be versioned separately.

However, having them as two separate projects does make it a bit more difficult for distribution. I'm writing this post to show you how I recently added support for distributing our application as 2 WARs or 1 WAR using the power of Maven, war overlays and the UrlRewriteFilter.

Project Setup
First of all, we have several different Maven modules, but the most important ones are as follows:

product-services
product-client
product-integration-tests

Of course, our modules aren't really named "product", but you get the point. The services project is really just a WAR project with Spring Security configured. It depends on other JAR modules that the services exist in. The client project is a GWT WAR that has a proxy servlet defined in its web.xml that makes it easier to develop. It also contains some UrlRewrite configuration that allows GWT Log's Remote Logging feature to work. The proxy servlet is something we don't want to ship with our product, so we have a separate web.xml for production vs. development. We do the substitution using the maven-war-plugin:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-war-plugin</artifactId>
    <version>2.0.2</version>
    <configuration>
        <!-- Production web.xml -->
        <webXml>src/main/resources/web.xml</webXml>
        <warSourceDirectory>war</warSourceDirectory>
        <!-- Exclude everything but urlrewrite JAR -->
        <warSourceExcludes>
            WEB-INF/lib/aop**,WEB-INF/lib/commons-**,WEB-INF/lib/gin-**,
            WEB-INF/lib/guice-**,WEB-INF/lib/gwt-**,WEB-INF/lib/gxt-**,
            WEB-INF/lib/junit-**
        </warSourceExcludes>
    </configuration>
</plugin>

I could exclude WEB-INF/lib/** and WEB-INF/classes/**, but in my particular project, we still want UrlRewrite in standalone mode, and we have some i18n properties files in WEB-INF/classes that are served up for Selenium tests.

With this configuration, we have a services WAR and a client WAR that can be installed and used by clients. To collapse them into one and make it possible to ship a single war, I turned to our product-integration-tests module. This module contains Selenium tests that test both types of distributions.

Merging 2 WARs into 1
The most important thing in the product-integration-tests module is that it creates a single WAR. First of all, it uses <packaging>war</packaging> to make this possible. The rest is done using the following 3 steps.

1. Its dependencies include the client and servlet WARs (and Selenium RC for testing).

<dependencies>
    <dependency>
        <groupId>com.company.app</groupId>
        <artifactId>product-services</artifactId>
        <version>1.0-SNAPSHOT</version>
        <type>war</type>
    </dependency>
    <dependency>
        <groupId>com.company.app</groupId>
        <artifactId>product-client</artifactId>
        <version>1.0-SNAPSHOT</version>
        <type>war</type>
    </dependency>
    <dependency>
        <groupId>org.seleniumhq.selenium.client-drivers</groupId>
        <artifactId>selenium-java-client-driver</artifactId>
        <version>1.0.1</version>
        <scope>test</scope>
    </dependency>
</dependencies>

2. The WAR created excludes the "integration-tests" part of the name:

<build>
    <finalName>product-${project.version}</finalName>
    ...
</build>

3. WAR overlays are configured so the everything in the client's WEB-INF directory is excluded from the merged WAR.

<plugin>
    <artifactId>maven-war-plugin</artifactId>
    <configuration>
        <!-- http://maven.apache.org/plugins/maven-war-plugin/overlays.html -->
        <overlays>
            <overlay>
                <groupId>com.company.app</groupId>
                <artifactId>product-services</artifactId>
                <excludes>
                    <!-- TODO: Rename to api.html (this is the Enunciate-generated documentation) -->
                    <exclude>index.html</exclude>
                </excludes>
            </overlay>
            <!-- No server needed in product-client -->
            <overlay>
                <groupId>com.company.app</groupId>
                <artifactId>product-client</artifactId>
                <excludes>
                    <exclude>WEB-INF/**</exclude>
                </excludes>
            </overlay>
            <!-- Only include META-INF/context.xml to set the ROOT path -->
            <overlay>
                <excludes>
                    <exclude>WEB-INF/**</exclude>
                </excludes>
            </overlay>
        </overlays>
    </configuration>
</plugin>

That's it! Using this configuration, it's possible to distribute a Maven-based SOFEA project as single or multiple WARs. However, there are some nuances.

One thing you might notice is the reference to META-INF/context.xml in the overlays configuration. This subtly highlights one issue I experienced when merging the WARs. In our GWT client, we're using URLs that point to our services at /product-services/*. This works in development (via a proxy servlet) and when the WARs are installed separately - as long as the services WAR is installed at /product-services. However, when they're merged, a little URL rewriting needs to happen. To do this, I added the UrlRewriteFilter to the product-services module and configured a simple rule.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCENGINE urlrewrite PUBLIC "-//tuckey.org//DTD UrlRewrite 3.0//EN"
        "http://tuckey.org/res/dtds/urlrewrite3.0.dtd">

<urlrewrite use-query-string="true">
    <!-- Used when services are merged into WAR with GWT client -->
    <rule>
        <from>^/product-services/(.*)$</from>
        <to type="forward">/$1</to>
    </rule>
</urlrewrite>

Because the services URLs point to the root (/product-services), the merged WAR has to be installed as the ROOT application. When you're using Cargo with Tomcat and want to deploy to ROOT, you have to have a META-INF/context.xml with a path="" reference (ref: CARGO-516).

<Context path=""/>

It is possible to change the URLs in the client to be relative, but this gets seems to get messy when you're using separate WARs. When using relative URLs, I found I had to do solution using cross-context forwarding to get the results I wanted. Using a redirect instead of a forward worked, but resulted in the client talking to the server twice (once to get redirected, a second time for the actual call). Cross-context forwarding is supported by the UrlRewriteFilter and Tomcat, but I'm not sure WebSphere or WebLogic support it. The best solution is probably to change the URLs dynamically at runtime, possibly using some sort of deferred binding technique.

Testing with Cargo and Selenium
Once I had everything merged, I wanted to configure Cargo and Selenium to allow testing both distribution types. If I installed all 3 wars at the same time, the "product-services" WAR would be used by both the product-client.war and the product.war, so I had to use profiles to allow installing the single merged WAR or both WARs. Below is the profile I used for starting Cargo, deploying the merged WAR, starting Selenium RC and running Selenium tests.

<properties>
    <cargo.container>tomcat6x</cargo.container>
    <cargo.container.url>
        http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.20/bin/apache-tomcat-6.0.20.zip
    </cargo.container.url>
    <cargo.host>localhost</cargo.host>
    <cargo.port>23433</cargo.port>
    <cargo.wait>false</cargo.wait>
    <cargo.version>1.0</cargo.version>

    <!-- *safari and *iexplore are additional options -->
    <selenium.browser>*firefox</selenium.browser>
</properties>
...
<profile>
    <id>itest-bamboo</id>
    <activation>
        <activeByDefault>false</activeByDefault>
    </activation>
    <build>
        <plugins>
            <plugin>
                <groupId>org.codehaus.cargo</groupId>
                <artifactId>cargo-maven2-plugin</artifactId>
                <version>${cargo.version}</version>
                <configuration>
                    <wait>${cargo.wait}</wait>
                    <container>
                        <containerId>${cargo.container}</containerId>
                        <log>${project.build.directory}/${cargo.container}/cargo.log</log>
                        <zipUrlInstaller>
                            <url>${cargo.container.url}</url>
                            <installDir>${installDir}</installDir>
                        </zipUrlInstaller>
                    </container>
                    <configuration>
                        <home>${project.build.directory}/${cargo.container}/container</home>
                        <properties>
                            <cargo.hostname>${cargo.host}</cargo.hostname>
                            <cargo.servlet.port>${cargo.port}</cargo.servlet.port>
                        </properties>
                        <!-- Deploy as ROOT since XHR requests are made to /product-services -->
                        <deployables>
                            <deployable>
                                <properties>
                                    <context>ROOT</context>
                                </properties>
                            </deployable>
                        </deployables>
                    </configuration>
                </configuration>
                <executions>
                    <execution>
                        <id>start-container</id>
                        <phase>pre-integration-test</phase>
                        <goals>
                            <goal>start</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>stop-container</id>
                        <phase>post-integration-test</phase>
                        <goals>
                            <goal>stop</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>selenium-maven-plugin</artifactId>
                <version>1.0</version>
                <executions>
                    <execution>
                        <phase>pre-integration-test</phase>
                        <goals>
                            <goal>start-server</goal>
                        </goals>
                        <configuration>
                            <background>true</background>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <artifactId>maven-surefire-plugin</artifactId>
                <executions>
                    <execution>
                        <phase>integration-test</phase>
                        <goals>
                            <goal>test</goal>
                        </goals>
                        <configuration>
                            <excludes>
                                <exclude>none</exclude>
                            </excludes>
                            <includes>
                                <include>**/*SeleniumTest.java</include>
                            </includes>
                            <systemProperties>
                                <property>
                                    <name>selenium.browser</name>
                                    <value>${selenium.browser}</value>
                                </property>
                                <property>
                                    <name>cargo.port</name>
                                    <value>${cargo.port}</value>
                                </property>
                            </systemProperties>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</profile>

This profile is run by our Bamboo nightly tests with mvn install -Pitest-bamboo. The 2nd profile I added doesn't install the project's WAR, but instead installs the two separate WARs. Running mvn install -Pitest-bamboo,multiple-wars executes the Selenium tests against the multi-WAR distribution.

<profile>
    <id>multiple-wars</id>
    <activation>
        <activeByDefault>false</activeByDefault>
    </activation>
    <build>
        <plugins>
            <plugin>
                <groupId>org.codehaus.cargo</groupId>
                <artifactId>cargo-maven2-plugin</artifactId>
                <version>${cargo.version}</version>
                <configuration>
                    <configuration>
                        <home>${project.build.directory}/${cargo.container}/container</home>
                        <properties>
                            <cargo.hostname>${cargo.host}</cargo.hostname>
                            <cargo.servlet.port>${cargo.port}</cargo.servlet.port>
                        </properties>
                        <deployables>
                            <deployable>
                                <groupId>com.company.app</groupId>
                                <artifactId>product-client</artifactId>
                                <pingURL>http://${cargo.host}:${cargo.port}/product-client/index.html</pingURL>
                                <type>war</type>
                                <properties>
                                    <context>/product-client</context>
                                </properties>
                            </deployable>
                            <deployable>
                                <groupId>com.company.app</groupId>
                                <artifactId>product-services</artifactId>
                                <pingURL>
                                    http://${cargo.host}:${cargo.port}/project-services/index.jspx
                                </pingURL>
                                <type>war</type>
                                <properties>
                                    <context>/product-services</context>
                                </properties>
                            </deployable>
                        </deployables>
                    </configuration>
                </configuration>
            </plugin>
        </plugins>
    </build>
</profile>

I won't be including any information on authoring Selenium tests because there's already many good references. I encourage you to checkout the following if you're looking for Selenium testing techniques.

Summary
This article has shown you how I used Maven, war overlays and the UrlRewriteFilter to allow create different distributions of a SOFEA application. I'm still not sure which packaging (1 WAR vs. 2) mechanism is best, but it's nice to know there's options. If you package and distribute SOFEA applications, I'd love to hear about your experience in this area.

Building LinkedIn's Next Generation Architecture with OSGi by Yan Pujante

2008-10-20T11:20:09-06:00

This week, I'm attending the Colorado Software Summit in Keystone, Colorado. Below are my notes from an OSGi at LinkedIn presentation I attended by Yan Pujante.

LinkedIn was created in March of 2003. Today there's close to 30M members. In the first 6 months, there was 60,000 members that signed up. Now, 1 million sign up every 2-3 weeks. LinkedIn is profitable with 5 revenue lines and there's 400 employees in Mountain View.

Technologies: 2 datacenters (~600 machines). SOA with Java, Tomcat, Spring Framework, Oracle, MySQL, Servlets, JSP, Cloud/Graph, OSGi. Development is done on Mac OS X, production is on Solaris.

The biggest challenges for LinkedIn are:

Growing Engineering Team on a monolithic code base (it's modular, but only one source tree)
Growing Product Team wanting more and more features faster
Growing Operations Team deploying more and more servers

Front-end has many BL services in one webapp (in Tomcat). The backend is many wars in 5 containers (in Jetty) with 1 service per WAR. Production and Development environments are very different. Total services in backend is close to 100, front-end has 30-40.

Container Challenges
1 WAR with N services does not scale for developers (conflicts, monolithic). N wars with 1 service does not scale for containers (no shared JARs). You can add containers, but there's only 12GB of RAM available.

Upgrading back-end service to new version requires downtime (hardware load-balancer does not account for version). Upgrading front-end service to new version requires redeploy. Adding new backend services is also painful because there's lots of configuration (load-balancer, IPs, etc.).

Is there a solution to all these issues? Yan believes that OSGi is a good solution. OSGi stands for Open Services Gateway initiative. Today that term doesn't really mean anything. Today it's a spec with several implementations: Equinox (Eclipse), Knoplerfish and Felix (Apache).

OSGi has some really, really good features. These include smart class loading (multiple versions of JARs is OK), it's highly dynamic (deploy/undeploy built-in), it has a service registry and is highly extensible/configurable. An OSGi bundle is simply a JAR file with an OSGi manifest.

In LinkedIn's current architecture, services are exported with Spring/RPC and services in same WAR can see each other. The problem with this architecture comes to light when you want to move services to a 2nd web container. You cannot share JARs and can't talk directly to the other web app. With OSGi, the bundles (JARs) are shared, services are shared and bundles can be dynamically replaced. OSGi solves the container challenge.

One thing missing from OSGi is allowing services to live on multiple containers. To solve this, LinkedIn has developed a way to have Distributed OSGi. Multicast is used to see what's going on in other containers. Remote servers use the OSGi lifecycle and create dynamic proxies to export services using Spring RPC over HTTP. Then this service is registered in the service registry on the local server.

With Distributed OSGi, there's no more N-1 / 1-N problem. Libraries and services can be shared in one container (memory footprint is much smaller). Services can be shared across containers. The location of the services is transparent to the clients. There's no more configuration to change when adding/removing/moving services. This architecture allows the software to be the load balancer instead of using a hardware load balancer.

Unfortunately, everything is not perfect. OSGi has quite a few problems. OSGi is great, but the tooling is not quite there yet. Not every library is a bundle and many JARs doesn't have OSGi manifests. OSGi was designed for embedded devices and using it for the server-side is very recent (but very active).

OSGi is pretty low-level, but there is some work being done to hide the complexity. Spring DM helps, as do vendor containers. SpringSource has Spring dm Server, Sun has GlassFish, and Paremus has Infiniflow. OSGi is container centric, but next version will add distributed OSGi, but will have no support for load-balancing.

Another big OSGi issue is version management. If you specify version=1.0.0, it means [1.0.0, ∞]. You should at least use version=[1.0.0,2.0.0]. When using OSGi, you have to be careful and follow something similar to Apache's APR Project Versioning Guidelines so that you can easily identify release compatibility.

At LinkedIn, the OSGi implementation is progressing, but there's still a lot of work to do. First of all, a bundle repository needs to be created. Ivy and bnd is used to generate bundles. Containers are being evaluated and Infiniflow is most likely the one that will be used. LinkedIn Spring (an enhanced version of Spring) and SCA will be used to deploy composites. Work on load-balancing and distribution is in progress as is work on tooling and build integration (Sigil from Paremus).

In conclusion, LinkedIn will definitely use OSGi but they'll do their best to hide the complexity from the build system and from developers. For more information on OSGi at LinkedIn, stay tuned to the LinkedIn Engineering Blog. Yan has promised to blog about some of the challenges LinkedIn has faced and how he's fixed them.

Update: Yan has posted his presentation on the LinkedIn Engineering Blog.

Jetty and Resin closing in on Tomcat's popularity

2008-04-11T08:42:48-06:00

From Greg Wilkin's Jetty Improves in Netcraft survey (again):

As with most open source projects, it's very hard to get a measure of who/how/where/why Jetty is being used a deployed. Downloads long ago became meaningless with the advent of many available bundling and distribution channels. The Netcraft Web Survey is one good measure, as it scans the internet and identifies which server sites run. In the results released April 2008, Jetty is identified for 278,501 public server, which is 80% of the market share of our closest "competitor" tomcat (identified as coyote in the survey). Jetty is currently 12th in the league table of identified servers of all types and will be top 10 in 6 months if the current trajectory continues.

If you look at the Netcraft numbers, you might also notice that Resin isn't far behind Jetty. If you look at the Indeed Job Trends graphs for the three, there seems to be some interesting information there too. The first graph is absolute and the second is relative.

tomcat, resin, jetty Job Trends

tomcat jobs - resin jobs - jetty jobs

tomcat, resin, jetty Job Trends

tomcat jobs - resin jobs - jetty jobs

If you're using Spring Dynamic Modules to deploy a web application, which server do you think is better? Both Tomcat 6 and Jetty 6 seem to work just fine in Equinox.

GlassFish 2 vs. Tomcat 6

2007-09-19T16:55:31-06:00

In Switched, Dave says:

Now that Glassfish V2 is out I'm switching from Tomcat to Glassfish for all of my development. It's more than fast enough. With Glassfish on my MacBook Pro, Roller restart time is about 8 seconds compared to 16 with Tomcat. And the quality is high; the admin console, the asadmin command-line utility and the docs are all excellent. The dog food is surprisingly tasty ;-)

I did some brief and very non-scientific performance comparisons myself:

Startup Time with no applications deployed:

Tomcat 6: 3 seconds
GlassFish 2: 8 seconds

Startup Time with AppFuse 2.0 (Struts + Hibernate version) as a WAR

Tomcat 6: 15 seconds
GlassFish 2: 16 seconds

Environment:

JAVA_OPTS="-Xms768M -Xmx768M -XX:PermSize=512m -XX:MaxPermSize=512m -Djava.awt.headless=true -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:+UseConcMarkSweepGC -server"
OS X 10.4.10, 2.2 GHz Intel Core 2 Duo, 4 GB 667 MHz DDR2 SDRAM

Since this was a very non-scientific experiment, it's possible the last two are actually the same. It's strange that Dave is seeing Roller startup twice as fast on GlassFish. Maybe they've done some Roller deployment optimization?

I realize startup times aren't that important. However, as Dave mentions, they (and context reloading) can be extremely important when developing.

Update: I got to thinking that Dave is probably referring to context reloading. Here's a comparison of how long it takes for both servers to pick up a new WAR (and start the application) when it's dropped into their autodeploy directories.

Tomcat 6: 14-16 seconds
GlassFish 2: 9 seconds

The strange thing about Tomcat is it takes 6-8 seconds to recognize a new WAR has been deployed. Does Tomcat have a polling increment that can be increased during development?

Regardless, it's impressive that the GlassFish guys have made things that much faster for developers. Nice work folks!

These days, I try to use mvn jetty:run on projects. Then I don't have to worry about deploying, just save and wait for the reload. Time to wait for AppFuse 2.0 to reload using the Maven Jetty Plugin (version 6.1.5)? 7 seconds. Of course, it'd be nice if I could somehow get this down to 1 or 2 seconds.

Maybe Dave should use the Maven integration for Roller to decrease his reload times. ;-)

Proposed Tomcat Enhancement: Add flag to escape JSP's EL by default

2007-09-19T16:29:11-06:00

I posted the following to the Tomcat Developers mailing list. Unfortunately, it didn't get any responses, which means (to me) that no one cares about this feature. I guess the good thing is they didn't veto it.

Hello all,

I'm working for a client that's using a proprietary Servlet/JSP-based framework that runs on Tomcat. They have their own custom JSP compiler and they're looking to move to a standard JSP compiler. One of the things their compiler supports is automatic escaping of XML in expressions. For example, ${foo} would be escaped so <body> -> <body>. JSP EL does not do this. It *doesn't* escape by default and instead requires you to wrap your expressions with <c:out/> if you want escaping.

I'd like to ask what developers think about adding a flag (similar to trimSpaces in conf/web.xml) that allows users to change the escaping behavior from false to true?

I think this is a good option to have as it allows security-conscious organizations to paranoid and escape all content by default.

Thanks,

Matt

Related: http://raibledesigns.com/rd/entry/java_web_frameworks_and_xss

What do you think? Is there anything wrong with adding this (optional) feature to Tomcat? Enhancing security is a good thing - right?

Update: I've entered an enhancement request for this feature and attached a patch.

Jetty 6.x versus Tomcat 6.x

2007-08-15T09:50:17-06:00

An AppFuse user asks:

Has anyone done any performance benchmarking between Jetty 6.x and Tomcat 6.x to see which one is better for production use in terms of scalability, performance and ease-of-use? I'm gearing towards Jetty 6.1 but want to hear other's opinions first.

I admit, I completely changed the wording in this quote to make it more readable.

Most of the companies I've worked with in recent years have been using Tomcat (very successfully) in production. However, I also know the Contegix and JavaLobby guys continue to swear by Resin for the most part. What's your opinion?

IMHO, I don't think it really matters - they're all good enough for production use.

Configuring Tomcat for Production

2007-04-16T16:05:03-06:00

From The Atlassian Developer Blog:

You appear to be running Tomcat in development mode. (http://tomcat.apache.org/tomcat-5.5-doc/jasper-howto.html)

This means that source of every JSP is held in memory to provide detailed messages in the event of an error. If you have large JSPs this will hurt. It looks like this accounts for 50MB+

From Tomcat's Production Configuration documentation:

The main JSP optimization which can be done is precompilation of JSPs. However, this might not be possible (for example, when using the jsp-property-group feature) or practical, in which case the configuration of the Jasper servlet becomes critical.

When using Jasper 2 in a production Tomcat server you should consider making the following changes from the default configuration.

development - To disable on access checks for JSP pages compilation set this to false.
genStringAsCharArray - To generate slightly more efficient char arrays, set this to true.
modificationTestInterval - If development has to be set to true for any reason (such as dynamic generation of JSPs), setting this to a high value will improve performance a lot.
trimSpaces - To remove useless bytes from the response, set this to true.

Seems like good information to know.

Mixing Apache HTTP Server, mod_rewrite and mod_jk

2007-04-16T12:07:10-06:00

I'm trying to configure Apache and Tomcat to work with a desired architecture for doing A/B Testing on my current project. Our basic idea is that we'll deploy entirely new WAR files when we have a test, then use the magic of Apache's mod_rewrite, mod_jk and possible the UrlRewriteFilter to keep the URLs somewhat consistent between version A and version B. Here's some questions I have for those folks who might've done this before:

Is it possible to use Apache's mod_rewrite to map http://www.domain.com/?v=1 to http://www.domain.com/1 (this allows us to have two different applications/wars served up to the same domain).
If #1 is possible, what's the RewriteRule for allowing the parameter to be anywhere in the query string, but still allowing the target to use it as the context name?
Is it possible to use something in the WAR (likely the UrlRewriteFilter) to produce HTML that has rewritten links (i.e. http://www.domain.com/?id=1 in the WAR whose context is 1)?

In other words, can Apache forward to the correct app going in, and can that app rewrite its URLs so those same URLs are used when going out?

I believe this is all possible. However, I am having difficulty getting mod_jk to allow mod_rewrite to be processed first. If I have the following in httpd.conf, it seems like htdocs/.htaccess gets bypassed.

JkMount /* loadbalancer

Is it possible to configure Apache/mod_jk so WARs can hang off the root, but still use mod_rewrite? If not, the only solution I can think of is to use UrlRewriteFilter in the WAR to forward to another context when a "v" parameter is in the URL. Currently, the UrlRewriteFilter doesn't allow forwarding to another context. The good news is the Servlet API allows it. I got it working in Tomcat (with crossContext enabled) and wrote a patch for the UrlRewriteFilter.

Anyone out there have experience doing A/B Testing in a Java webapp? If so, did you try to disguise the URLs for the different versions?

Update: I've got a bit of this working. The magic formula seems to be don't try to hang things off the root - use mod_rewrite to make things appear to hang off the root.

First of all, I posted a message similar to this post to the tomcat-user mailing list. Before I did so, I discovered mod_proxy_ajp, which happens to look like the successor to mod_jk. AFAICT, it doesn't allow fine-grained rules (i.e. only serve up *.jsp and *.do from Tomcat), so I'll stick with mod_jk for now.

Rather than proxying all root-level requests to Tomcat, I changed my JkMount to expect all Tomcat applications to have a common prefix. For example, "app".

JkMount /app* loadbalancer

This allows me to create RewriteRules in htdocs/.htaccess to detect the "v" parameter and forward to Tomcat.

RewriteEngine On

RewriteCond     %{QUERY_STRING}     ^v=(.*)$
RewriteRule     ^(.*)$              /app%1/ [L]

This isn't that robust as adding another parameter causes the forward to fail. However, it does successfully forward http://localhost/?v=1 to /app1 on Tomcat and http://localhost/?v=2 to /app2 on Tomcat.

What about when ?v=3 is passed in? There's no /app3 installed on Tomcat, so Tomcat's ROOT application will be hit. Using the UrlRewriteFilter, I installed a root application (which we'll likely need anyway) with the following rule:

    <rule>
        <from>^/app(.*)$</from>
        <to type="forward">/</to>
    </rule>

So I've solved problem #1: Using URL parameters to serve up different web applications. To solve the second issue (webapps should rewrite their URLs to delete their context path), I found two solutions:

Use mod_proxy_html. Sounds reasonable, but requires the use of mod_proxy.
Use the UrlRewriteFilter and outbound-rules.

Since I'm using mod_jk, #2 is the reasonable choice. I added the following link in my /app1/index.jsp:

<a href="<c:url value="/products.jsp"/>">link to products</a>

By default, this gets written out as http://localhost/app1/products.jsp. To change it to http://localhost/products.jsp?v=1, I added the following to urlrewrite.xml:

    <outbound-rule>
        <from>^/app1/(.*)$</from>
        <to>/$1?v=1</to>
    </outbound-rule>

This produces the desired effect, except that when I click on the link, a new session is created every time. AFAICT, I probably need to do something with cookies so the jsessionid cookie is set for the proper path.

Not bad for a day's work. Only 2 questions remain:

What's a more robust RewriteRule that doesn't care about other parameters being passed in?
What do I need to do so new sessions aren't created when using an outbound-rule?

It's entirely possible that mod_proxy_ajp with mod_rewrite_html is best tool for this. Can mod_proxy handle wildcards like JkMount can? I've heard it's faster than mod_jk, so it probably warrants further investigation.

Update 2: I achieved the desired result using mod_rewrite, mod_jk and the UrlRewriteFilter (for outgoing links). Here's what I put in htdocs/.htaccess (on Apache):

RewriteEngine On

# http://domain/?v=1 --> http://domain/app1/?v=1
RewriteCond     %{QUERY_STRING}     v=([^&]+)
RewriteRule     ^(.*)$              /app%1/$1 [L]

# http://domain --> http://domain/app (default ROOT in Tomcat)
RewriteRule     ^$                  /app/ [L]

And in the urlrewrite.xml of each webapp:

   <outbound-rule>
       <from>^/app([0-9])/([A-Za-z0-9]+)\.([A-Za-z0-9]+)$</from>
       <to>/$2.$3?v=$1</to>
   </outbound-rule>

   <outbound-rule>
       <from>^/app([0-9])/([A-Za-z0-9]+)\.([A-Za-z0-9]+)\?(.*)$</from>
       <to>/$2.$3?$4&amp;v=$1</to>
   </outbound-rule>

Next I'll try to see if I can get it all working with mod_proxy_ajp and mod_proxy_html. Anyone know the equivalent of "JkMount /app*" when using LocationMatch with mod_proxy?

Upgrading to Tomcat 6

2007-03-02T00:44:46-07:00

Erik did it, so I tried it as well. This site is now running Tomcat 6.0.10 and it has to be the least painful major Tomcat upgrade I've ever done. By major, I mean upgrading from one version number (5.5.17) to the next. Apparently, no XML files changed (like they did from 4.1.x -> 5.0.x -> 5.5.x) because I was able to copy over conf/server.xml and conf/Catalina/** without any issues. The only change I had to make was to copy commons-logging.jar from Roller's WEB-INF/lib to JSPWiki's.

I have seen a couple of the following errors in my log files since I upgraded, so if you see any strange behavior, please let me know.

2-Mar-2007 12:36:10 AM org.apache.tomcat.util.http.Parameters processParameters
WARNING: Parameters: Character decoding failed. Parameter skipped.
java.io.CharConversionException: EOF
        at org.apache.tomcat.util.buf.UDecoder.convert(UDecoder.java:83)
        at org.apache.tomcat.util.buf.UDecoder.convert(UDecoder.java:49)
        at org.apache.tomcat.util.http.Parameters.urlDecode(Parameters.java:410)
        at org.apache.tomcat.util.http.Parameters.processParameters(Parameters.java:392)
        at org.apache.tomcat.util.http.Parameters.processParameters(Parameters.java:508)
        at org.apache.tomcat.util.http.Parameters.handleQueryParameters(Parameters.java:266)
        at org.apache.catalina.connector.Request.parseParameters(Request.java:2404)
        at org.apache.catalina.connector.Request.getParameterValues(Request.java:1089)
        at org.apache.catalina.connector.RequestFacade.getParameterValues(RequestFacade.java:396)
        at javax.servlet.ServletRequestWrapper.getParameterValues(ServletRequestWrapper.java:189)
        at org.acegisecurity.wrapper.SavedRequestAwareWrapper.getParameter(SavedRequestAwareWrapper.java:325)
        at org.apache.roller.ui.rendering.velocity.deprecated.OldPageRequest.(OldPageRequest.java:164)
        at org.apache.roller.ui.rendering.velocity.deprecated.RedirectServlet.figurePageRedirect(RedirectServlet.java:285)
        at org.apache.roller.ui.rendering.velocity.deprecated.RedirectServlet.doGet(RedirectServlet.java:131)

I tested AppFuse 2.0 on Tomcat 6.0.10 earlier today and impressed that 1) Cargo worked perfectly and 2) most of the web frameworks worked. Which one didn't? You guessed it - good ol' JSF. That's OK though, the JSF version of AppFuse (MyFaces 1.1.5 with Facelets 1.1.11) doesn't work with Jetty 6.1.1 either. The good news is I found a workaround - removing the el-api dependency from my pom.xml makes it work on both.

    <dependency>
        <groupId>javax.el</groupId>
        <artifactId>el-api</artifactId>
        <version>1.2</version>
    </dependency>

If I remove this dependency, everything works fine on Tomcat 6.0.10 and Jetty 6.1.1. Unfortunately, it seems this dependency is needed for Tomcat 5.x. Hopefully some fancy stuff with Maven profiles can fix this incompatibility.