Making Code Generation Smarter with Maven
As you might've read in my last entry, I recently started a new gig with Overstock.com. On my first day, I was quickly immersed into the development process by joining the Conversion Team. The Conversion Team is responsible for developing the checkout UI and handling payments from customers. I quickly discovered Overstock was mostly a Linux + Eclipse Shop and did my best to get my favorite Mac + IntelliJ + JRebel installed and configured. Thanks to my new Team Lead, I was able to get everything up and running the first day, as well as checkin my first contribution: making mvn jetty:run work so I didn't have to use my IDE to deploy to Tomcat.
In setting up my environment, I couldn't help but notice running jetty:run took quite a while to run each time. Specifically, the build process took 45 seconds to start executing the Jetty plugin, then another 23 seconds to startup after that. The first suspicious thing I noticed was that the UI templates were being re-generated and compiled on each execution. The UI Templating Framework at Overstock is Jamon, and is described as follows:
Jamon is a text template engine for Java, useful for generating dynamic HTML, XML, or any text-based content. In a typical Model-View-Controller architecture, Jamon clearly is aimed at the View (or presentation) layer.
Because it is compiled to non-reflective Java code, and statically type-checked, Jamon is ideally suited to support refactoring of template-based UI applications. Using mock objects -like functionality, Jamon also facilitates unit testing of the controller and view.
To generate .java files from .jamon templates, we use the Jamon Plugin for Maven. Remembering that the Maven Compiler Plugin has an incremental-compile feature, I turned to its source code to find out how to implement this in the Jamon plugin. I was pleasantly surprised to find the StaleSourceScanner. This class allows you to easily compare two files to see if the source needs to re-examined for generation or compilation.
I noticed the Jamon Plugin had the following code to figure out which files it should generate into .java files:
private List<File> accumulateSources(File p_templateSourceDir)
{
final List<File> result = new ArrayList<File>();
if (p_templateSourceDir == null)
{
return result;
}
for (File f : p_templateSourceDir.listFiles())
{
if (f.isDirectory())
{
result.addAll(accumulateSources(f));
}
else if (f.getName().toLowerCase(Locale.US).endsWith(".jamon"))
{
String filePath = f.getPath();
// FIXME !?
String basePath = templateSourceDir().getAbsoluteFile().toString();
result.add(new File(filePath.substring(basePath.length() + 1)));
}
}
return result;
}
I changed it to be smarter and only generate changed templates with the following code:
private List<File> accumulateSources(File p_templateSourceDir) throws MojoExecutionException
{
final List<File> result = new ArrayList<File>();
if (p_templateSourceDir == null)
{
return result;
}
SourceInclusionScanner scanner = getSourceInclusionScanner( staleMillis );
SourceMapping mapping = new SuffixMapping( ".jamon", ".java");
scanner.addSourceMapping( mapping );
final Set<File> staleFiles = new LinkedHashSet<File>();
for (File f : p_templateSourceDir.listFiles())
{
if (!f.isDirectory())
{
continue;
}
try
{
staleFiles.addAll( scanner.getIncludedSources(f.getParentFile(), templateOutputDir()));
}
catch ( InclusionScanException e )
{
throw new MojoExecutionException(
"Error scanning source root: \'" + p_templateSourceDir.getPath()
+ "\' " + "for stale files to recompile.", e );
}
}
// Trim root path from file paths
for (File file : staleFiles) {
String filePath = file.getPath();
String basePath = templateSourceDir().getAbsoluteFile().toString();
result.add(new File(filePath.substring(basePath.length() + 1)));
}
}
This method references a getSourceInclusionScanner() method, which is implemented as follows:
protected SourceInclusionScanner getSourceInclusionScanner( int staleMillis )
{
SourceInclusionScanner scanner;
if ( includes.isEmpty() && excludes.isEmpty() )
{
scanner = new StaleSourceScanner( staleMillis );
}
else
{
if ( includes.isEmpty() )
{
includes.add( "**/*.jamon" );
}
scanner = new StaleSourceScanner( staleMillis, includes, excludes );
}
return scanner;
}
If you're using Jamon and its Maven Plugin, you can view my patch at SourceForge. If you're looking to include this functionality in your project, I invite you to look at the code I learned from in the Maven Compiler's AbstractCompilerMojo class.
After making this change, I was able to reduce the build execution time by over 50%. Now it takes 20 seconds to hit the Jetty plugin and 42 seconds to finishing starting. Of course, in an ideal world, I'd like to get this down to 20 seconds or less. Strangely enough, the easiest way to do this seems to be simple: use Linux.
On the Linux desktop they provided me, it takes 12 seconds to hit the Jetty plugin and 23 seconds to finish starting. I'd like to think this is a hardware thing, but it only get 20% faster on OS X when using an 8GB RAM + SSD machine (vs. a 4GB + 5400 drive). Since Overstock has provided me with a 4GB MacBook Pro, I'm considering installing Ubuntu on it, just to see what the difference is.
In related news, Overstock.com is looking to hire a whole bunch of Java Developers this year. The pictures of the new Provo office look pretty sweet. Of course, you can also work at HQ, which is a mere 25 minutes from some of the best skiing in the world. Personally, I think Colorado's powder is better, but I can't argue with the convenience of no traffic. In addition to full-time gigs, they've started hiring more remote contractors like myself, so they pretty much have something for everyone. So if you love Java, like to get some turns in before work, and aren't an asshole - you should
and I'll try to hook you up.
Update: After writing this post, I received an email from Neil Hartner questioning these numbers. Basically, he was able to get his MacBook Pro to run just as fast as Linux. Turns out, the reason my Mac was so much slower was because JRebel was configured in my MAVEN_OPTS. JRebel's FAQ does state the following:
Does JRebel make the server start up slower?
JRebel needs to do more work on startup (search more places for classes and resources, instrument classes, etc), so some slowdown can be expected. If it's larger than 50% please contact [email protected].
Since it's right around 50% slower, I guess there's no reason to call them. My guess is the best thing to do is remove JRebel from MAVEN_OPTS, but have an alias that can enable it, or simply run it from your IDE.