Matt RaibleMatt Raible is a Web Architecture Consultant specializing in open source frameworks.

10+ YEARS


Over 10 years ago, I wrote my first blog post. Since then, I've authored books, had kids, traveled the world, found Trish and blogged about it all.

Making Code Generation Smarter with Maven

As you might've read in my last entry, I recently started a new gig with Overstock.com. On my first day, I was quickly immersed into the development process by joining the Conversion Team. The Conversion Team is responsible for developing the checkout UI and handling payments from customers. I quickly discovered Overstock was mostly a Linux + Eclipse Shop and did my best to get my favorite Mac + IntelliJ + JRebel installed and configured. Thanks to my new Team Lead, I was able to get everything up and running the first day, as well as checkin my first contribution: making mvn jetty:run work so I didn't have to use my IDE to deploy to Tomcat.

In setting up my environment, I couldn't help but notice running jetty:run took quite a while to run each time. Specifically, the build process took 45 seconds to start executing the Jetty plugin, then another 23 seconds to startup after that. The first suspicious thing I noticed was that the UI templates were being re-generated and compiled on each execution. The UI Templating Framework at Overstock is Jamon, and is described as follows:

Jamon is a text template engine for Java, useful for generating dynamic HTML, XML, or any text-based content. In a typical Model-View-Controller architecture, Jamon clearly is aimed at the View (or presentation) layer.

Because it is compiled to non-reflective Java code, and statically type-checked, Jamon is ideally suited to support refactoring of template-based UI applications. Using mock objects -like functionality, Jamon also facilitates unit testing of the controller and view.

To generate .java files from .jamon templates, we use the Jamon Plugin for Maven. Remembering that the Maven Compiler Plugin has an incremental-compile feature, I turned to its source code to find out how to implement this in the Jamon plugin. I was pleasantly surprised to find the StaleSourceScanner. This class allows you to easily compare two files to see if the source needs to re-examined for generation or compilation.

I noticed the Jamon Plugin had the following code to figure out which files it should generate into .java files:

private List<File> accumulateSources(File p_templateSourceDir)
{
  final List<File> result = new ArrayList<File>();
  if (p_templateSourceDir == null)
  {
    return result;
  }
  for (File f : p_templateSourceDir.listFiles())
  {
    if (f.isDirectory())
    {
      result.addAll(accumulateSources(f));
    }
    else if (f.getName().toLowerCase(Locale.US).endsWith(".jamon"))
    {
      String filePath = f.getPath();
       // FIXME !?
      String basePath = templateSourceDir().getAbsoluteFile().toString();
      result.add(new File(filePath.substring(basePath.length() + 1)));
    }
  }
  return result;
}

I changed it to be smarter and only generate changed templates with the following code:

private List<File> accumulateSources(File p_templateSourceDir) throws MojoExecutionException
{
  final List<File> result = new ArrayList<File>();
  if (p_templateSourceDir == null)
  {
    return result;
  }
  SourceInclusionScanner scanner = getSourceInclusionScanner( staleMillis );
  SourceMapping mapping = new SuffixMapping( ".jamon", ".java");

  scanner.addSourceMapping( mapping );

  final Set<File> staleFiles = new LinkedHashSet<File>();

  for (File f : p_templateSourceDir.listFiles())
  {
    if (!f.isDirectory())
    {
      continue;
    }

    try
    {
      staleFiles.addAll( scanner.getIncludedSources(f.getParentFile(), templateOutputDir()));
    }
    catch ( InclusionScanException e )
    {
      throw new MojoExecutionException(
        "Error scanning source root: \'" + p_templateSourceDir.getPath()
          + "\' " + "for stale files to recompile.", e );
    }
  }

  // Trim root path from file paths
  for (File file : staleFiles) {
    String filePath = file.getPath();
    String basePath = templateSourceDir().getAbsoluteFile().toString();
    result.add(new File(filePath.substring(basePath.length() + 1)));
  }
}

This method references a getSourceInclusionScanner() method, which is implemented as follows:

protected SourceInclusionScanner getSourceInclusionScanner( int staleMillis )
{
  SourceInclusionScanner scanner;

  if ( includes.isEmpty() && excludes.isEmpty() )
  {
      scanner = new StaleSourceScanner( staleMillis );
  }
  else
  {
      if ( includes.isEmpty() )
      {
          includes.add( "**/*.jamon" );
      }
      scanner = new StaleSourceScanner( staleMillis, includes, excludes );
  }

  return scanner;
}

If you're using Jamon and its Maven Plugin, you can view my patch at SourceForge. If you're looking to include this functionality in your project, I invite you to look at the code I learned from in the Maven Compiler's AbstractCompilerMojo class.

After making this change, I was able to reduce the build execution time by over 50%. Now it takes 20 seconds to hit the Jetty plugin and 42 seconds to finishing starting. Of course, in an ideal world, I'd like to get this down to 20 seconds or less. Strangely enough, the easiest way to do this seems to be simple: use Linux.

On the Linux desktop they provided me, it takes 12 seconds to hit the Jetty plugin and 23 seconds to finish starting. I'd like to think this is a hardware thing, but it only get 20% faster on OS X when using an 8GB RAM + SSD machine (vs. a 4GB + 5400 drive). Since Overstock has provided me with a 4GB MacBook Pro, I'm considering installing Ubuntu on it, just to see what the difference is.

Sun over the Snowbird In related news, Overstock.com is looking to hire a whole bunch of Java Developers this year. The pictures of the new Provo office look pretty sweet. Of course, you can also work at HQ, which is a mere 25 minutes from some of the best skiing in the world. Personally, I think Colorado's powder is better, but I can't argue with the convenience of no traffic. In addition to full-time gigs, they've started hiring more remote contractors like myself, so they pretty much have something for everyone. So if you love Java, like to get some turns in before work, and aren't an asshole - you should and I'll try to hook you up.

Update: After writing this post, I received an email from Neil Hartner questioning these numbers. Basically, he was able to get his MacBook Pro to run just as fast as Linux. Turns out, the reason my Mac was so much slower was because JRebel was configured in my MAVEN_OPTS. JRebel's FAQ does state the following:

Does JRebel make the server start up slower?
JRebel needs to do more work on startup (search more places for classes and resources, instrument classes, etc), so some slowdown can be expected. If it's larger than 50% please contact support@zeroturnaround.com.

Since it's right around 50% slower, I guess there's no reason to call them. My guess is the best thing to do is remove JRebel from MAVEN_OPTS, but have an alias that can enable it, or simply run it from your IDE.

Posted in Java at Jan 21 2011, 03:26:43 PM MST 7 Comments
Comments:

I've been using Jamon for a while and find it to be the best templating engine, overall. I'm really surprised it's basically unknown, especially as it is basically the only statically typed templating engine around.

Posted by Moandji Ezana on January 23, 2011 at 10:41 AM MST #

Thanks for informing me of Jamon (never heard of that).

Choosing IntelliJ + JRebel over Eclipse + plugins is easy to understand but not choosing an expensive Mac over Linux. Could you tell me your reasons for choosing Mac?

Posted by Thai Dang Vu on January 24, 2011 at 08:59 AM MST #

Thai Dang Vu - I prefer Mac over Linux, mostly for the laptop experience and UI. Tried Linux on the desktop for a couple years (2004-2005), but never liked it enough to replace OS X. As far as expense is concerned, I'm willing to pay for good tools. ;)

Posted by Matt Raible on January 24, 2011 at 02:40 PM MST #

Having just completed a year working for Overstock.com and working with Jamon, I can definitely see the advantages, but I always felt that most of those advantages could be gained from pre-compiling JSPs as well. The thing that makes it work well for Overstock's purposes is the fact that their site is primarily REST and stateless. There was never a need for a request or session object, which was frustrating as a JSP/JSF developer. One disadvantage is that there are many instances of Java code living on those Jamon templates and pages. As a JSP dev, you have to forget all you know about separating the concerns by keeping scriptlet code out of the pages.

Posted by Fred Olivieri on January 25, 2011 at 01:18 PM MST #

"I always felt that most of those advantages could be gained from pre-compiling JSPs as well"

To me, the main advantage is static typing, which JSPs just can't have.

"There was never a need for a request or session object"

I use the request and session objects, but use dependency injection patterns to make it painless: for each request, I inject the context object into controllers or into a Templates class, which itself has methods for specific templates.

"One disadvantage is that there are many instances of Java code living on those Jamon templates and pages. As a JSP dev, you have to forget all you know about separating the concerns by keeping scriptlet code out of the pages."

I think it depends what that code is. JSP Custom Tags are merely an inconvenient way of putting Java code in templates. Well-factored Jamon templates and applications don't mix business logic and views, but the view technology itself has nothing to do with that. The way to separate concerns is through thinking about design and architecture, not using a crippled system.

Posted by Moandji Ezana on January 25, 2011 at 04:06 PM MST #

why Jamon and not FreeMarker, for instance?

Posted by Anton on January 12, 2012 at 07:05 AM MST #

I believe Overstock liked Jamon better because of its compile-time checking, rather than runtime like FreeMarker. Personally, I'm not sold on the idea, but LinkedIn had a similar templating framework when I worked there.

Posted by Matt Raible on January 12, 2012 at 07:07 AM MST #

Post a Comment:
  • HTML Syntax: Allowed