Raible Designs

AppFuse Light 2.2.1 Released!

In December, the AppFuse Team released 2.2.1. Right before that release, I decided to wait on enhancing its "light" modules, a.k.a. AppFuse Light. I'm glad I did, because it took some effort to get jQuery and Bootstrap integrated, as well as to make it more secure.

The good news is AppFuse Light 2.2.1 is released and it's sitting out on the Central Repository. This release is a refactoring of all archetypes to be up-to-date with the AppFuse 2.2.1 release. This means Java 7 compatibility, Servlet 3, Bootstrap/jQuery integration, Tapestry 5.3.6 upgrade and security improvements. I integrated Bootstrap and jQuery using WebJars Servlet 3 support since it was simple and straightforward.

You can create projects using AppFuse's light archetypes using a command such as the following:

mvn archetype:generate -B -DarchetypeGroupId=org.appfuse.archetypes 
  -DarchetypeArtifactId=appfuse-light-spring-freemarker-archetype -DarchetypeVersion=2.2.1 
  -DgroupId=com.mycompany -DartifactId=myproject 

The list of archetypes is as follows:

• appfuse-light-jsf-archetype
• appfuse-light-spring-archetype
• appfuse-light-spring-freemarker-archetype
• appfuse-light-spring-security-archetype
• appfuse-light-stripes-archetype
• appfuse-light-struts-archetype
• appfuse-light-tapestry-archetype
• appfuse-light-wicket-archetype

The QuickStart Guide will help you get setup and demos are available at the following links:

• JSF 2 + Hibernate
• Spring MVC 3 + Hibernate
• Spring MVC 3 + FreeMarker + Hibernate
• Spring MVC 3 + Spring Security + Hibernate
• Stripes + Hibernate
• Struts 2 + Hibernate
• Tapestry 5 + Hibernate
• Wicket + Hibernate

If you have questions about AppFuse, we invite you to ask them on the users mailing list or tweet using #appfuse.

For those enjoying Bootstrap in your apps, I encourage you to check out {wrap}bootstrap and Bootswatch.

AppFuse 2.2.1 Released!

The AppFuse Team is pleased to announce the release of AppFuse 2.2.1. This release includes upgrades to all dependencies to bring them up-to-date with their latest releases. Most notable are Hibernate 4, Struts 2.3.7, Apache CXF 2.7.0 and Spring Security 3.1.3. In addition, we've integrated HTML5, Twitter Bootstrap, jQuery and replaced Compass with Hibernate Search. Last but not least, we've added full support for Java 7 and integrated many security improvements. For more details on specific changes see the release notes.

What is AppFuse?
AppFuse is a full-stack framework for building web applications on the JVM. It was originally developed to eliminate the ramp-up time when building new web applications. Over the years, it has matured into a very testable and secure system for creating Java-based webapps.

Demos for this release can be viewed at http://demo.appfuse.org. Please see the QuickStart Guide to get started with this release.

A number of blog posts were written about features that went into this release while it was being developed:

• Improving AppFuse's PageSpeed with Apache
• New Liquibase and wro4j Tutorials from J. García
• AppFuse News: GitHub, Hibernate Search and The Future
• Refreshing AppFuse's UI with Twitter Bootstrap
• Upgrading AppFuse to Spring Security 3.1 and Spring 3.1
• Integrating OAuth with AppFuse and its REST API

If you have questions about AppFuse, please read the FAQ or join the mailing list. If you find any issues, please report them on the users mailing list.

Thanks to everyone for their help contributing patches, writing documentation and participating on the mailing lists.

We greatly appreciate the help from our sponsors, particularly Atlassian, Contegix and JetBrains. We highly recommend using the new IntelliJ IDEA 12 for developing web applications.

Deploying Java and Play Framework Apps to the Cloud with James Ward

Yesterday, I attended James Ward's presentation on Deploying Java & Play Framework Apps to the Cloud at Devoxx. I arrived a bit late, but still managed to get there in time to see a lot of demos and learn more about Heroku. Below are my notes from James's talk.

When I arrived, James was doing a demo using Spring Roo. He was using Roo's Petclinic sample app and showed us how you could use Git to create a local repository of the new project and install Heroku's command line tool. From there, he ran the following command to create a new application on Heroku.

heroku create -s cedar

The Cedar Stack is what supports Java, Scala and Play Framework. It's the 3rd generation stack for Heroku. The command above created two endpoints, one for HTTP and one for Git. It picks from a list of randomly generated names, which all seem to have some humor in them. James ended up with "electric-sword-8877" for this demo.

From there, he ran git push heroku master to deploy the project to Heroku. Unfortunately, this resulted in a login error and there was an akward moment where we all thought the Demo Gods were angry. However, James was able to resolve this by using Heroku's sharing feature with the following command.

heroku sharing:add jw@heroku.com

For Java projects, Heroku looks for a pom.xml file in the root directory and runs a Maven build on project. All the dependencies get downloaded on the cloud rather than put them into a WAR and requiring you to upload a large WAR file. You don't have to upload your source code to Heroku; James did it for the sake of the demo because it was faster.

After the build finishes, it creates a slug file. This file contains everything Heroku needs to run your application.

Next, James showed a demo of the running application and added a new Pet through its UI. Then he scaled it to two servers using the following command:

heroku scale web=2

He proved this was working by running heroku ps, which showed there were two running processes. He showed the app again, but noted that the record he added was missing. This is because when it started up a new dyno, Hibernate created the schema again and deleted all records. To fix, James changed Hibernate to only update the schema instead of create a new one. If you're a Hibernate user, you know this is as simple as changing:

hibernate.hbm2ddl.auto=create

to:

hibernate.hbm2ddl.auto=update

After committing this change, James redeployed using Git.

git push heroku master

The slug file got built again and Heroku deployed the new slug onto both dynos, automatically load balancing the app across two servers. James then ran heroku logs to see the logs of his dynos and prove that a request to his app's HTTP endpoint made requests to both dynos. The logging is powered by Logplex and you can read about how it works in the article Heroku Gets Sweet Logging.

James mentioned that Roo has a Heroku plugin, but after watching his talk and searching a bit on the internet, it seems it's just the jetty-runner setup as described in Getting Started with Spring MVC Hibernate on Heroku/Cedar.

What about autoscaling? There are some 3rd party tools that do this. Heroku's Management infrastructure has APIs that these tools talk too. Heroku hasn't built autoscaling into the platform because they don't know where the bottlenecks are in your application.

Heroku = Polyglot + PaaS + Cloud Components. It supports Ruby, node.js, Java, Clojure, Play and Scala and they're working on native Grails and Gradle support. There's currently 534,374 apps running on Heroku.

Heroku is a cloud application platform and there's 5 different components.

1. Instant deployment
2. HTTP Routing / Load Balancing
3. Elastic Polyglot Runtime
4. Management & Logging
5. Component as a Service Ecosystem

For instant deployment, it's a pretty simple process:

• You add files to a git repo
• You provision the app on Heroku (heroku create)
• You upload the files to Heroku (git push heroku master)
• Heroku runs the build and assembles a "slug" file
• Heroku starts a "dyno"
• Heroku copies the "slug" to the "dyno"
• Heroku starts the web application

Most apps will contain a Procfile that contains information about how to run the web process. For Spring Roo, it has:

web: java $JAVA_OPTS -jar target/dependency/jetty-runner.jar --port $PORT target/*.war

So how does Heroku decide what application server to use? It doesn't, you do. You need to get your application server into the slug file. The easiest way to do this is to specify your application server as a dependency in your pom.xml. In the Roo example, James uses the maven-dependency-plugin to get the jetty-runner dependency and copy it to the target directory. On Heroku, you bring your application server with you.

Heroku gives you 750 free dyno hours per app, per month. For developers, it's very easy to get started and use. Once you extend past one dyno, it's $.05 per dyno hour, which works out to around $30/month. It's only when you want to scale beyond one dyno where you get charged by Heroku, no matter how much data you transfer. Scalatest is running on Heroku. It has one dyno and is doing fine with that. Bill Venners doesn't have to pay anything for it.

java.herokuapp.com is a site James created that allows you to clone example apps and get started quickly with Heroku's Cedar Stack.

For HTTP Routing, Heroku uses an Erlang-based routing system to route all the HTTP requests across your dynos. Heroku doesn't support sticky sessions. Distributed session management does not work well, because it does not scale well. Heroku recommends you use a stateless web architecture or move your state into something like memcached. Jetty has (in the latest version) the ability to automatically serialize your session into a Mongo system. This works fine on Heroku. The problem with this is if you have 2 dynos running, each request can hit a different dyno and get different session state. Hence the recommendation for an external storage mechanism that can synchronize between dynos.

Heroku has native support for the Play framework. To detect Play applications, it look for a conf/application.conf file. You don't need to have a Procfile in your root directory because Heroku knows how to start a Play application.

At this point, James created a new Play application, created a new Heroku app (he got "young-night-7104" this time) and pushed it to Heroku. He created a simple model object, a controller to allow adding new data and then wrote some jQuery to show new records via Ajax and JSON. He also showed how to configure the application to talk to Heroku's PostgreSQL database using the DATABASE_URL environment variable. He explained how you can use the heroku config command to see your environment variables.

The reason they use environment variables is so Heroku can update DATABASE_URL (and other variables) without having to call up all their customers and have them change them in their source code.

Play on Heroku supports Scala if you create your app with Scala. Play 2.0 uses Scala, Akka and SBT. Heroku added support for SBT a couple month ago, so everything will work just fine.

Heroku also supports Scala, detecting it by looking for the build.sbt file in the root directory. Heroku supports SBT 0.11.0 and it builds the 'stage' task. It currently does not support Lift because Lift uses an older version of SBT and because it's a very stateful framework that would require sticky sessions. Use Play, BlueEyes or Scalatra if you want Scala on Heroku.

Heroku has addons for adding functionality to your application, including Custom DNS, HTTPS, Amazon RDS, NoSQL and many more. They're also working on making their add-on and management APIs available via Java, so you'll (hopefully) be able to use them from your IDE in the future.

From there, James showed us how Heroku keeps slug files around so you can do rollbacks with heroku rollback. He also showed how you can use:

heroku run "your bash command"

Summary
I attended James's talk because he's a good friend, but also because I've been using Heroku to host my latest adventures with Play, Scala, CoffeeScript and Jade. I'm glad I attended because I learned some good tips and tricks and more about how Heroku works.

Heroku seems like a great development tool to me. In my experience, it's been really nice to have instant deployments using Git. In fact, I've created a 'push' alias so I can push to my project's repo and heroku at the same time.

alias push='git push origin master && git push heroku master'

I'd like to see more organizations embrace something like Heroku for developers. It'd be great if everyone had their own sandbox that business owners and product managers could see. I can't help but think this would be awesome for demos, prototyping, etc.

There were some other talks I wanted to attend at the same time, particularly Martin Odersky's What's in store for Scala? and WWW: World Wide Wait? A Performance Comparison of Java Web Frameworks. The WWW talk has posted their presentation but I'm sure it'd be more fun to watch.

It's pretty awesome that all the talks from Devoxx 2011 will be up on Parleys.com soon.

Update: James has posted his slides from this talk.

AppFuse 2.1 Released!

The AppFuse Team is pleased to announce the release of AppFuse 2.1. This release includes upgrades to all dependencies to bring them up-to-date with their latest releases. Most notable are JPA 2, JSF 2, Tapestry 5 and Spring 3. In addition, we've migrated from XFire to CXF and enabled REST for web services. There's even a new appfuse-ws archetype that leverages Enunciate to generate web service endpoints, documentation and downloadable clients. This release fixes many issues with archetypes, improving startup time and allowing jetty:run to be used for quick turnaround while developing. For more details on specific changes see the release notes.

What is AppFuse?
AppFuse is an open source project and application that uses open source frameworks to help you develop Web applications with Java quickly and efficiently. It was originally developed to eliminate the ramp-up time when building new web applications. At its core, AppFuse is a project skeleton, similar to the one that's created by your IDE when you click through a wizard to create a new web project. If you use JRebel with IntelliJ, you can achieve zero-turnaround in your project and develop features without restarting the server.

Release Details
Archetypes now include all the source for the web modules so using jetty:run and your IDE will work much smoother now. The backend is still embedded in JARs, enabling you to choose with persistence framework (Hibernate, iBATIS or JPA) you'd like to use. If you want to modify the source for that, add the core classes to your project or run "appfuse:full-source".

AppFuse comes in a number of different flavors. It offers "light", "basic" and "modular" and archetypes. Light archetypes use an embedded H2 database and contain a simple CRUD example. Light archetypes allow code generation and full-source features, but do not currently support Stripes or Wicket. Basic archetypes have web services using CXF, authentication from Spring Security and features including signup, login, file upload and CSS theming. Modular archetypes are similar to basic archetypes, except they have multiple modules which allows you to separate your services from your web project.

AppFuse provides archetypes for JSF, Spring MVC, Struts 2 and Tapestry 5. The light archetypes are available for these frameworks, as well as for Spring MVC + FreeMarker, Stripes and Wicket. You can see demos of these archetypes at http://demo.appfuse.org.

For information on creating a new project, please see the QuickStart Guide.

If you have questions about AppFuse, please read the FAQ or join the user mailing list. If you find any issues, please report them on the mailing list or create an issue in JIRA.

Thanks to everyone for their help contributing patches, writing documentation and participating on the mailing lists.

We greatly appreciate the help from our sponsors, particularly Atlassian, Contegix and JetBrains. Atlassian and Contegix are especially awesome: Atlassian has donated licenses to all its products and Contegix has donated an entire server to the AppFuse project.

Integration Testing with HTTP, HTTPS and Maven

Earlier this week, I was tasked with getting automated integration tests working in my project at Overstock.com. By automated, I mean that ability to run "mvn install" and have the following process cycled through:

• Start a container
• Deploy the application
• Run all integration tests
• Stop the container

Since it makes sense for integration tests to run in Maven's integration-test phase, I first configured the maven-surefire-plugin to skip tests in the test phase and execute them in the integration-test phase. I used the <id>default-phase</id> syntax to override the plugins' usual behavior.

<plugin>
  <artifactId>maven-surefire-plugin</artifactId>
  <executions>
    <execution>
      <id>default-test</id>
      <configuration>
        <excludes>
          <exclude>**/*Test*.java</exclude>
        </excludes>
      </configuration>
    </execution>
    <execution>
      <id>default-integration-test</id>
      <phase>integration-test</phase>
      <goals>
        <goal>test</goal>
      </goals>
      <configuration>
        <includes>
          <include>**/*Test.java</include>
        </includes>
        <excludes>
          <exclude>none</exclude>
          <exclude>**/TestCase.java</exclude>
        </excludes>
      </configuration>
    </execution>
  </executions>
</plugin>

After I had this working, I moved onto getting the container started and stopped properly. In the past, I've done this using Cargo and it's always worked well for me. Apart from the usual setup I use in AppFuse archetypes (example pom.xml), I added a couple additional items:

• Added <timeout>180000</timeout> so the container would wait up to 3 minutes for the WAR to deploy.
• In configuration/properties, specified <context.path>ROOT</context.path> so the app would deploy at the / context path.
• In configuration/properties, specified <cargo.protocol>https</cargo.protocol> since many existing unit tests made requests to secure resources.

I started by using Cargo with Tomcat and had to create certificate keystore in order to get Tomcat to start with SSL enabled. After getting it to start, I found the tests failed with the following errors in the logs:

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: 
PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: 
unable to find valid certification path to requested target
	at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1649)

Co-workers told me this was easily solved by adding my 'untrusted' cert to my JVM keystore. Once all this was working, I thought I was good to go, but found that some tests were still failing. The failures turned out to be because they were talking to http and https was the only protocol enabled. After doing some research, I discovered that Cargo doesn't support starting on both http and https ports.

So back to the drawing board I went. I ended up turning to the maven-jetty-plugin and the tomcat-maven-plugin to get the functionality I was looking for. I also automated the certificate keystore generation using the keytool-maven-plugin. Below is the extremely-verbose 95-line profiles section of my pom.xml that allows either container to be used.

Sidenote: I wonder how this same setup would look using Gradle?

<profiles>
  <profile>
    <id>jetty</id>
    <activation>
      <activeByDefault>true</activeByDefault>
    </activation>
    <build>
      <plugins>
        <plugin>
          <groupId>org.mortbay.jetty</groupId>
          <artifactId>maven-jetty-plugin</artifactId>
          <version>6.1.26</version>
          <configuration>
            <contextPath>/</contextPath>
            <connectors>
              <connector implementation="org.mortbay.jetty.nio.SelectChannelConnector">
                <!-- forwarded == true interprets x-forwarded-* headers -->
                <!-- http://docs.codehaus.org/display/JETTY/Configuring+mod_proxy -->
                <forwarded>true</forwarded>
                <port>8080</port>
                <maxIdleTime>60000</maxIdleTime>
              </connector>
              <connector implementation="org.mortbay.jetty.security.SslSocketConnector">
                <forwarded>true</forwarded>
                <port>8443</port>
                <maxIdleTime>60000</maxIdleTime>
                <keystore>${project.build.directory}/ssl.keystore</keystore>
                <password>overstock</password>
                <keyPassword>overstock</keyPassword>
              </connector>
            </connectors>
            <stopKey>overstock</stopKey>
            <stopPort>9999</stopPort>
          </configuration>
          <executions>
            <execution>
              <id>start-jetty</id>
              <phase>pre-integration-test</phase>
              <goals>
                <goal>run-war</goal>
              </goals>
              <configuration>
                <daemon>true</daemon>
              </configuration>
            </execution>
            <execution>
              <id>stop-jetty</id>
              <phase>post-integration-test</phase>
              <goals>
                <goal>stop</goal>
              </goals>
            </execution>
          </executions>
        </plugin>
      </plugins>
    </build>
  </profile>
  <profile>
    <id>tomcat</id>
    <build>
      <plugins>
        <plugin>
          <groupId>org.codehaus.mojo</groupId>
          <artifactId>tomcat-maven-plugin</artifactId>
          <version>1.1</version>
          <configuration>
            <addContextWarDependencies>true</addContextWarDependencies>
            <fork>true</fork>
            <path>/</path>
            <port>8080</port>
            <httpsPort>8443</httpsPort>
            <keystoreFile>${project.build.directory}/ssl.keystore</keystoreFile>
            <keystorePass>overstock</keystorePass>
          </configuration>
          <executions>
            <execution>
              <id>start-tomcat</id>
              <phase>pre-integration-test</phase>
              <goals>
                <goal>run-war</goal>
              </goals>
            </execution>
            <execution>
              <id>stop-tomcat</id>
              <phase>post-integration-test</phase>
              <goals>
                <goal>shutdown</goal>
              </goals>
            </execution>
          </executions>
        </plugin>
      </plugins>
    </build>
  </profile>
</profiles>

With this setup in place, I was able to automate running our integration tests by simply typing "mvn install" (for Jetty) or "mvn install -Ptomcat" (for Tomcat). For running in Hudson, it's possible I'll have to further enhance things to randomize the port and pass that into tests as a system property. The build-helper-maven-plugin and its reserve-network-port goal is a nice way to do this. Note: if you want to run more than one instance of Tomcat at a time, you might have to randomize the ajp and rmi ports to avoid collisions.

The final thing I encountered was our app didn't shutdown gracefully. Luckily, this was fixed in a newer version of our core framework and upgrading fixed the problem. Here's the explanation from an architect on the core framework team.

The hanging problem was caused by the way the framework internally aggregated statistics related to database connection usage and page response times. The aggregation runs on a separate thread but not as a daemon thread. Previously, the aggregation threads weren't being terminated on shutdown so the JVM would hang waiting for them to finish. In the new frameworks, the aggregation threads are terminated on shutdown.

Hopefully this post helps you test your secure and unsecure applications at the same time. At the same time, I'm hoping it motivates the Cargo developers to add simultaneous http and https support.

Update: In the comments, Ron Piterman recommended I use the Maven Failsafe Plugin because its designed to run integration tests while Surefire Plugin is for unit tests. I changed my configuration to the following and everything still passes. Thanks Ron!

<plugin>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>2.7.2</version>
  <configuration>
    <skipTests>true</skipTests>
  </configuration>
</plugin>
<plugin>
  <artifactId>maven-failsafe-plugin</artifactId>
  <version>2.7.2</version>
  <configuration>
    <includes>
      <include>**/*Test.java</include>
    </includes>
    <excludes>
      <exclude>**/TestCase.java</exclude>
    </excludes>
  </configuration>
  <executions>
    <execution>
      <id>integration-test</id>
      <phase>integration-test</phase>
      <goals>
        <goal>integration-test</goal>
      </goals>
    </execution>
    <execution>
      <id>verify</id>
      <phase>verify</phase>
      <goals>
        <goal>verify</goal>
      </goals>
    </execution>
  </executions>
</plugin>

Update 2: In addition to application changes to solve hanging issues, I also had to change my Jetty Plugin configuration to use a different SSL connector implementation. This also required adding the jetty-sslengine dependency, which has been renamed to jetty-ssl for Jetty 7.

<connector implementation="org.mortbay.jetty.security.SslSelectChannelConnector">
...
<dependencies>
  <dependency>
    <groupId>org.mortbay.jetty</groupId>
    <artifactId>jetty-sslengine</artifactId>
    <version>6.1.26</version>
  </dependency>
</dependencies>

Making Code Generation Smarter with Maven

As you might've read in my last entry, I recently started a new gig with Overstock.com. On my first day, I was quickly immersed into the development process by joining the Conversion Team. The Conversion Team is responsible for developing the checkout UI and handling payments from customers. I quickly discovered Overstock was mostly a Linux + Eclipse Shop and did my best to get my favorite Mac + IntelliJ + JRebel installed and configured. Thanks to my new Team Lead, I was able to get everything up and running the first day, as well as checkin my first contribution: making mvn jetty:run work so I didn't have to use my IDE to deploy to Tomcat.

In setting up my environment, I couldn't help but notice running jetty:run took quite a while to run each time. Specifically, the build process took 45 seconds to start executing the Jetty plugin, then another 23 seconds to startup after that. The first suspicious thing I noticed was that the UI templates were being re-generated and compiled on each execution. The UI Templating Framework at Overstock is Jamon, and is described as follows:

Jamon is a text template engine for Java, useful for generating dynamic HTML, XML, or any text-based content. In a typical Model-View-Controller architecture, Jamon clearly is aimed at the View (or presentation) layer.

Because it is compiled to non-reflective Java code, and statically type-checked, Jamon is ideally suited to support refactoring of template-based UI applications. Using mock objects -like functionality, Jamon also facilitates unit testing of the controller and view.

To generate .java files from .jamon templates, we use the Jamon Plugin for Maven. Remembering that the Maven Compiler Plugin has an incremental-compile feature, I turned to its source code to find out how to implement this in the Jamon plugin. I was pleasantly surprised to find the StaleSourceScanner. This class allows you to easily compare two files to see if the source needs to re-examined for generation or compilation.

I noticed the Jamon Plugin had the following code to figure out which files it should generate into .java files:

private List<File> accumulateSources(File p_templateSourceDir)
{
  final List<File> result = new ArrayList<File>();
  if (p_templateSourceDir == null)
  {
    return result;
  }
  for (File f : p_templateSourceDir.listFiles())
  {
    if (f.isDirectory())
    {
      result.addAll(accumulateSources(f));
    }
    else if (f.getName().toLowerCase(Locale.US).endsWith(".jamon"))
    {
      String filePath = f.getPath();
       // FIXME !?
      String basePath = templateSourceDir().getAbsoluteFile().toString();
      result.add(new File(filePath.substring(basePath.length() + 1)));
    }
  }
  return result;
}

I changed it to be smarter and only generate changed templates with the following code:

private List<File> accumulateSources(File p_templateSourceDir) throws MojoExecutionException
{
  final List<File> result = new ArrayList<File>();
  if (p_templateSourceDir == null)
  {
    return result;
  }
  SourceInclusionScanner scanner = getSourceInclusionScanner( staleMillis );
  SourceMapping mapping = new SuffixMapping( ".jamon", ".java");

  scanner.addSourceMapping( mapping );

  final Set<File> staleFiles = new LinkedHashSet<File>();

  for (File f : p_templateSourceDir.listFiles())
  {
    if (!f.isDirectory())
    {
      continue;
    }

    try
    {
      staleFiles.addAll( scanner.getIncludedSources(f.getParentFile(), templateOutputDir()));
    }
    catch ( InclusionScanException e )
    {
      throw new MojoExecutionException(
        "Error scanning source root: \'" + p_templateSourceDir.getPath()
          + "\' " + "for stale files to recompile.", e );
    }
  }

  // Trim root path from file paths
  for (File file : staleFiles) {
    String filePath = file.getPath();
    String basePath = templateSourceDir().getAbsoluteFile().toString();
    result.add(new File(filePath.substring(basePath.length() + 1)));
  }
}

This method references a getSourceInclusionScanner() method, which is implemented as follows:

protected SourceInclusionScanner getSourceInclusionScanner( int staleMillis )
{
  SourceInclusionScanner scanner;

  if ( includes.isEmpty() && excludes.isEmpty() )
  {
      scanner = new StaleSourceScanner( staleMillis );
  }
  else
  {
      if ( includes.isEmpty() )
      {
          includes.add( "**/*.jamon" );
      }
      scanner = new StaleSourceScanner( staleMillis, includes, excludes );
  }

  return scanner;
}

If you're using Jamon and its Maven Plugin, you can view my patch at SourceForge. If you're looking to include this functionality in your project, I invite you to look at the code I learned from in the Maven Compiler's AbstractCompilerMojo class.

After making this change, I was able to reduce the build execution time by over 50%. Now it takes 20 seconds to hit the Jetty plugin and 42 seconds to finishing starting. Of course, in an ideal world, I'd like to get this down to 20 seconds or less. Strangely enough, the easiest way to do this seems to be simple: use Linux.

On the Linux desktop they provided me, it takes 12 seconds to hit the Jetty plugin and 23 seconds to finish starting. I'd like to think this is a hardware thing, but it only get 20% faster on OS X when using an 8GB RAM + SSD machine (vs. a 4GB + 5400 drive). Since Overstock has provided me with a 4GB MacBook Pro, I'm considering installing Ubuntu on it, just to see what the difference is.

In related news, Overstock.com is looking to hire a whole bunch of Java Developers this year. The pictures of the new Provo office look pretty sweet. Of course, you can also work at HQ, which is a mere 25 minutes from some of the best skiing in the world. Personally, I think Colorado's powder is better, but I can't argue with the convenience of no traffic. In addition to full-time gigs, they've started hiring more remote contractors like myself, so they pretty much have something for everyone. So if you love Java, like to get some turns in before work, and aren't an asshole - you should and I'll try to hook you up.

Update: After writing this post, I received an email from Neil Hartner questioning these numbers. Basically, he was able to get his MacBook Pro to run just as fast as Linux. Turns out, the reason my Mac was so much slower was because JRebel was configured in my MAVEN_OPTS. JRebel's FAQ does state the following:

Does JRebel make the server start up slower?
JRebel needs to do more work on startup (search more places for classes and resources, instrument classes, etc), so some slowdown can be expected. If it's larger than 50% please contact support@zeroturnaround.com.

Since it's right around 50% slower, I guess there's no reason to call them. My guess is the best thing to do is remove JRebel from MAVEN_OPTS, but have an alias that can enable it, or simply run it from your IDE.

AppFuse 2.1 Milestone 2 Released

I'm pleased to announce the 2nd milestone release of AppFuse 2.1. This release includes upgrades to all dependencies to bring them up-to-date with their latest releases. Most notable are Spring 3 and Struts 2.1. This release fixes many issues with archetypes and contains many improvements to support Maven 3. For more details on specific changes see the 2.1.0 M2 release notes.

What is AppFuse?
AppFuse is an open source project and application that uses open source frameworks to help you develop Web applications quickly and efficiently. It was originally developed to eliminate the ramp-up time when building new web applications. At its core, AppFuse is a project skeleton, similar to the one that's created by your IDE when you click through a wizard to create a new web project. If you use JRebel with AppFuse, you can achieve zero-turnaround in your project and develop features without restarting the server.

Release Details
Archetypes now include all the source for the web modules so using jetty:run and your IDE will work much smoother now. The backend is still embedded in JARs, enabling you to choose with persistence framework (Hibernate, iBATIS or JPA) you'd like to use. If you want to modify the source for that, add the core classes to your project or run "appfuse:full-source".

AppFuse comes in a number of different flavors. It offers "light", "basic" and "modular" and archetypes. Light archetypes use an embedded H2 database and contain a simple CRUD example. In the final 2.1.0 release, the light archetypes will allow code generation like the basic and modular archetypes. Basic archetypes have web services using CXF, authentication from Spring Security and features including signup, login, file upload and CSS theming. Modular archetypes are similar to basic archetypes, except they have multiple modules which allows you to separate your services from your web project.

AppFuse provides archetypes for JSF, Spring MVC, Struts 2 and Tapestry 5. The light archetypes are available for these frameworks, as well as for Spring MVC + FreeMarker, Stripes and Wicket.

Please note that this release does not contain updates to the documentation. Code generation will work, but it's likely that some content in the tutorials won't match. For example, you can use annotations (vs. XML) for Spring MVC and Tapestry is a whole new framework. I'll be working on documentation over the next several weeks in preparation for the 2.1 final release.

For information on creating a new project, please see the QuickStart Guide.

If you have questions about AppFuse, please read the FAQ or join the user mailing list. If you find bugs, please create an issue in JIRA.

Thanks to everyone for their help contributing patches, writing documentation and participating on the mailing lists.

Versioning Static Assets with UrlRewriteFilter

A few weeks ago, a co-worker sent me interesting email after talking with the Zoompf CEO at JSConf.

One interesting tip mentioned was how we querystring the version on our scripts and css. Apparently this doesn't always cache the way we expected it would (some proxies will never cache an asset if it has a querystring). The recommendation is to rev the filename itself.

This article explains how we implemented a "cache busting" system in our application with Maven and the UrlRewriteFilter. We originally used querystring in our implementation, but switched to filenames after reading Souders' recommendation. That part was figured out by my esteemed colleague Noah Paci.

Our Requirements

• Make the URL include a version number for each static asset URL (JS, CSS and SWF) that serves to expire a client's cache of the asset.
• Insert the version number into the application so the version number can be included in the URL.
• Use a random version number when in development mode (based on running without a packaged war) so that developers will not need to clear their browser cache when making changes to static resources. The random version number should match the production version number formats which is currently: x.y-SNAPSHOT-revisionNumber
• When running in production, the version number/cachebust is computed once (when a Filter is initialized). In development, a new cachebust is computed on each request.

In our app, we're using Maven, Spring and JSP, but the latter two don't really matter for the purposes of this discussion.

Implementation Steps
1. First we added the buildnumber-maven-plugin to our project's pom.xml so the build number is calculated from SVN.

<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>buildnumber-maven-plugin</artifactId>
    <version>1.0-beta-4</version>
    <executions>
        <execution>
            <phase>validate</phase>
            <goals>
                <goal>create</goal>
            </goals>
        </execution>
    </executions>
    <configuration>
        <doCheck>false</doCheck>
        <doUpdate>false</doUpdate>
        <providerImplementations>
            <svn>javasvn</svn>
        </providerImplementations>
    </configuration>
</plugin>

2. Next we used the maven-war-plugin to add these values to our WAR's MANIFEST.MF file.

<plugin>
    <artifactId>maven-war-plugin</artifactId>
    <version>2.0.2</version>
    <configuration>
        <archive>
            <manifest>
                <addDefaultImplementationEntries>true</addDefaultImplementationEntries>
            </manifest>
            <manifestEntries>
                <Implementation-Version>${project.version}</Implementation-Version>
                <Implementation-Build>${buildNumber}</Implementation-Build>
                <Implementation-Timestamp>${timestamp}</Implementation-Timestamp>
            </manifestEntries>
        </archive>
    </configuration>
</plugin>

3. Then we configured a Filter to read the values from this file on startup. If this file doesn't exist, a default version number of "1.0-SNAPSHOT-{random}" is used. Otherwise, the version is calculated as ${project.version}-${buildNumber}.

private String buildNumber = null;

...
@Override
public void initFilterBean() throws ServletException {
    try {
        InputStream is = 
            servletContext.getResourceAsStream("/META-INF/MANIFEST.MF");
        if (is == null) {
            log.warn("META-INF/MANIFEST.MF not found.");
        } else {
            Manifest mf = new Manifest();
            mf.read(is);
            Attributes atts = mf.getMainAttributes();
            buildNumber = atts.getValue("Implementation-Version") + "-" + atts.getValue("Implementation-Build");
            log.info("Application version set to: " + buildNumber);
        }
     } catch (IOException e) {
        log.error("I/O Exception reading manifest: " + e.getMessage());
     }
}

...

    // If there was a build number defined in the war, then use it for
    // the cache buster. Otherwise, assume we are in development mode 
    // and use a random cache buster so developers don't have to clear 
    // their browswer cache.
    requestVars.put("cachebust", buildNumber != null ? buildNumber : "1.0-SNAPSHOT-" + new Random().nextInt(100000));

4. We then used the "cachebust" variable and appended it to static asset URLs as indicated below.

<c:set var="version" scope="request" 
    value="${requestScope.requestConfig.cachebust}"/>
<c:set var="base" scope="request"
    value="${pageContext.request.contextPath}"/>

<link rel="stylesheet" type="text/css" 
    href="${base}/v/${version}/assets/css/style.css" media="all"/>

<script type="text/javascript" 
    src="${base}/v/${version}/compressed/jq.js"></script>

The injection of /v/[CACHEBUSTINGSTRING]/(assets|compressed) eventually has to map back to the actual asset (that does not include the two first elements of the URI). The application must remove these two elements to map back to the actual asset. To do this, we use the UrlRewriteFilter. The UrlRewriteFilter is used (instead of Apache's mod_rewrite) so when developers run locally (using mvn jetty:run) they don't have to configure Apache.

5. In our application, "/compressed/" is mapped to wro4j's WroFilter. In order to get UrlRewriteFilter and WroFilter to work with this setup, the WroFilter has to accept FORWARD and REQUEST dispatchers.

<filter-mapping>
    <filter-name>rewriteFilter</filter-name>
    <url-pattern>/*</url-pattern>
</filter-mapping>

<filter-mapping>
    <filter-name>WebResourceOptimizer</filter-name>
    <url-pattern>/compressed/*</url-pattern>
    <dispatcher>FORWARD</dispatcher>
    <dispatcher>REQUEST</dispatcher>
</filter-mapping>

Once this was configured, we added the following rules to our urlrewrite.xml to allow rewriting of any assets or compressed resource request back to its "correct" URL.

<rule match-type="regex">
    <from>^/v/[0-9A-Za-z_.\-]+/assets/(.*)$</from>
    <to>/assets/$1</to>
</rule>
<rule match-type="regex">
    <from>^/v/[0-9A-Za-z_.\-]+/compressed/(.*)$</from>
    <to>/compressed/$1</to>
</rule>
<rule>
    <from>/compressed/**</from>
    <to>/compressed/$1</to>
</rule>

Of course, you can also do this in Apache. This is what it might look like in your vhost.d file:

RewriteEngine    on
RewriteLogLevel  0!
RewriteLog       /srv/log/apache22/app_rewrite_log
RewriteRule      ^/v/[.A-Za-z0-9_-]+/assets/(.*) /assets/$1 [PT]
RewriteRule      ^/v/[.A-Za-z0-9_-]+/compressed/(.*) /compressed/$1 [PT]

Whether it's a good idea to implement this in Apache or using the UrlRewriteFilter is up for debate. If we're able to do this with the UrlRewriteFilter, the benefit of doing this at all in Apache is questionable, especially since it creates a duplicate of code.

What's New in Maven 3.0 with Matthew McCullough

Last night, I attended the Denver JUG meeting to hear some excellent talks by Matthew McCullough and Tim Berglund. I took notes during Matthew's talk, but my battery ran out before Tim's talk started. Below are my notes.

Matthew started out by described the differences between Maven 2 and Maven 3. As he began, he emphasized it wasn't a beginner talk, but mostly for existing Maven users that understand how to read a pom.xml and such.

The Roadmap
Commits to Maven 3 have been happening for the last 3 years. Matthew is not an employee of Sonatype, but he mentioned their name quite a bit in his talk. Sonatype has hired several committers (7 that Matthew knows of by name) that now work on Maven 3 full-time. For compatibility with Maven 2, the project has 450 integration tests and they test it against 100s of Maven 2 projects. Maven 3 has plugin classloader partitioning and a legacy simulation layer for old plugins.

The main improvement in Maven 3 is speed. It's been performance tuned to be 50% to 400% faster. Benchmarks (guaranteed by integration tests) include better: Disk I/O, Network I/O, CPU and Memory. Another new feature is extensibility so Maven is a better library rather than just a command-line tool. Now there's a library and APIs that you can use to do the things that Maven does. Plexus has been replaced with Guice and it's now much easier to embed Maven (Polyglot Maven and Maven Shell are examples of this).

Below are a number of other changes between Maven 2 and Maven 3.

• Syntax: pom.xml still uses <modelVersion>4.0.0</modelVersion> so it can be a drop-in replacement for Maven 2 projects.
• Validations: poms are heavily validated against common mistakes, warns when plugin versions are not specified (use mvn validate to see issues), blocks duplicate dependencies (examined in same POM only, conflict resolution used otherwise).
• Help URLs: wiki page URLs now shown for all error messages. One of the first Apache projects to do this.
• Removals: profiles.xml external file support removed, Maven 1.0 repository support removed <layout>legacy</layout> (it's been 5 years since any commits to Maven 1).
• Behavior: SNAPSHOTs always deployed with date-stamps, artifact resolution caching has been improved to do less checking (override with mvn <phase> -U).
• Plugins: version auto-selection favors RELEASEs over SNAPSHOTs (opposite for Maven 2), versions cannot be specified as RELEASE or LATEST, plugins only resolved from <pluginRepository> locations.
• See the Plugin Compatibility Matrix to see if your favorite plugins are compatible.

Maven 3 hopes to be a drop-in replacement for Maven 2, but non backwards-compatible changes will be happening in Maven 3.1. It's anticipated release is Q1 of 2011 and will likely contain the following features.

• "Mixins" for direct dependencies
• Site plugin takes over <reporting>
• Backwards compatibility by <modelVersion
• There's a good chance 3.1 breaks compatibility with legacy POMs

Another new thing in Maven 3 is Toolchain. Toolchain a common way to configure your JDK for multiple plugins. There are only a handful of plugins that are toolchain-enabled. User tool chain definitions are defined in ~/.m2/toolchains.xml. To use different toolchains (JDKs), you specify a vendor and version as part of your plugin configuration.

Maven Shell is a high performance console that's a Maven 3 add-on. It's hosted at GitHub to make community contributions easier. It goes on your command line and it offers syntax highlighting and context-sensitive help (by typing ? at the command prompt).

Another major improvement in Maven 3 is Polyglot Maven. Tools like Gant and Buildr have made Maven look ancient, but they've also given it a good challenge. Maven 3 is likely to leapfrog these tools because of its ability to use different languages for your build configuration. Currently, 6 languages are supported. Polyglot Maven is a super-set distribution of Maven 3. It's not shipped with Maven 3 core because it contains all the other language implementations and is quite large. Polyglot Maven also contains a translate tool that allows you to convert any-to-any language. It has a DSL framework with Macros and Lifecycle Hooks. Macros allows for more concise syntax.

After talking about Polyglot Maven a bit, Matthew shows us a demo translating pom.xml to pom.yaml and then running the build. After that, he showed us examples of what a pom looks like when defined in Clojure, Scala and Groovy. Someone asked about file parsing performance and Matthew said different languages would cause a single-digit performance difference as part of your build process. Personally, I can't help but think any non-XML parser would be faster than the XML parser.

In regards to m2eclipse, a new drop (0.10) occurred a few weeks ago and it's one of the highest quality releases to date. It has major refactoring and many performance improvements.

For sample Maven projects see Matthew's Maven Samples.

I very much enjoyed Matthew's talk, both because of his presentation techniques and because he had a lot of good information. While I've tried Maven 3 and Shell in the past, I've been newly inspired to start using them again on a daily basis.

Tim's talk on Decision Making was also excellent. The biggest things I learned were that conflict is good (idea-wise, not personal) and things to look out for between teams (fault lines). Hopefully both Tim and Matthew post their slides so I can link to them here.

Software Quality: The Quest for the Holy Grail?

This afternoon, I attended a session on software quality by Jesper Pedersen at TSSJS. Jesper is a Core Developer at JBoss by Red Hat. He's the project lead of JBoss JCA, Tattletale, Papaki and JBoss Profiler 2. He's also currently the chairman of the Boston JBoss User Group. In this session, Jesper hopes to define basic requirements for a development environment and offer ideas on how to clean up a messy project.

Is software quality a friend or a foe? By implementing software quality processes, are you introducing unnecessary overhead? Development platforms are different today. We write a lot more business-specific code. We depend on standard frameworks for the core functionality. We depend on vendors to implement the standards correctly. We also depend on vendors to provide the necessary integration layer.

Since the platform is now a large slice of the pie, we must make sure we know where the issue is located. We must have proper integration tests; we must manage dependencies. Today, we must treat dependencies as if they are part of the application.

Defining the platform for your project helps narrow down the dependencies for your project. The platform is composed of corporate guidelines, standards, vendors and backend systems that you have to integrate with. Documentation is key for a successful project. Key documents types: User Guide, Developer Guide, API Guide, Architect Design, Implementation and Test.

It helps to define a project-wide configuration management system. Define a code-formatting guide will add consistency in your source tree. Also make sure you have separate build, test and execution environments. Use a Maven repository for your dependencies; both to support your project's artifacts as well as vendor artifacts.

"Maven today is an industry standard." -- Jesper Pederson

Define your tool chain as you would for your application. Back your Maven repository with SCM tools like Subversion or Git. For testing, use JUnit (unit testing), Corbertura (test coverage) and Hudson (continuous integration). Furthermore, you can add Checkstyle and Findbugs to verify coding conventions and find basic issues with code.

For the build environment, you need to make sure your dependency metadata is correct. Also, make sure you use the best practices for your given build tool. For example, with Maven and Ivy, it's a good idea to extract the version numbers into a common area of your pom.xml/build.xml so you can easily view all the versions in use. After you've done this, you can externalize the version information from the environment. Watch out for transitive dependencies as they can be tricky. Make sure you know what gets pulled in. Use enforcers to approve/ban dependencies or turn it off (Ivy). You can also vote for MNG-2315. Finally, snapshot dependencies are evil: define your release process so that releases are easy.

What can you do if your project is already a mess? Signs that your project is a mess: you look at your platform as a big black box, you use different dependencies than your deployment platform or you don't have integration tests for sub-systems or dependencies. To fix this, you can use a tool to get an overview of the entire project. Enter Tattletale.

Tattletale can give you an overview of your dependencies (Ant and Maven integration). It's a static analysis tool that doesn't depend on metadata, scanning your class files instead. Using Tattletale, you can produce a number of reports about your dependencies, what they're dependent on and what's dependent on you.

To maintain the lead in your project, make sure to define a checklist for each stage of your development cycle. Do reviews on documentation, architecture, component design and code. Enforce your rules of your project with your build system.

Jesper's final thoughts:

• Maintaining dependencies for a software project can be a tricky task.
• Using an Open Source platform as the foundation will ease the investigation of issues and increase trust.
• Defining a project-wide tool chain is key.
• Enforce all the rules on the project (better up-front than "fixing it" afterwards)

As Dusty mentioned, this session has a lot of good (basic) information, but there wasn't much new stuff. My team is using many of the technologies and practices that Jesper has mentioned. I guess that's validation that we're doing it right. I've heard of Tattletale, but never had a need for it since I haven't been on any "messy" projects recently.