Nexus is a kick-ass Repository Manager

I started my current gig at the end of last year. I've been enjoying the work and especially the project infrastructure we've been using. We're using the usual suspects: JIRA, Confluence, Hudson and Subversion. We're also using a couple new ones, namely sventon and Nexus. For building, we're using Maven and Ivy (as a Grails plugin).

Nexus I'm writing this post to talk about Nexus and how much I've enjoyed using it. I like Nexus for two reasons: it's aesthetically pleasing and it's well-documented. Another reason I really dig it is because I haven't had to touch it since I first configured it. Software that just keeps on humming is always fun to work with.

Initially, I remember having some issues setting up repositories. I also remember solving them after learning how groups work.

In addition to on-the-job, I've started to use Nexus more and more in my open source life. With the help of Jason van Zyl, I recently moved AppFuse's repository to Sonatype's oss.sonatype.org. I also noticed there's a Nexus instance for Apache projects. If that's not enough, you can get Nexus Pro free if you're an open source project.

Personally, the open source version of Nexus seems good enough for me. While the Staging Suite looks nice, I think it's possible to do a lot of similar things with good communication. After all, it's not going to free you from having to wrestle with the maven-release-plugin.

Next week, I'm helping to polish and document our entire release process (from dev → qa → production). If you have any advice on how to best perform releases with Maven, Grails and/or Nexus, I'd love to hear about it. My goal is extreme efficiency so releases can be done very quickly and with minimal effort.

Posted in Java at Mar 05 2009, 11:59:02 PM MST 13 Comments

GWTTestSuite makes builds faster, but requires JUnit 4.1

Earlier this week, I spent some time implementing GWTTestSuite to speed up my project's build process. In Hudson, the project was taking around 15 minutes to build, locally it was only taking 5 minutes for mvn test. In IDEA, I could run all the tests in under a minute. While 15 minutes isn't a long time for a build to execute, a co-worker expressed some concern:

Does Maven have to run GWT test and individual Java processes? (See target/gwtTest/*.sh) This arrangement and the overhead of JVM launches is another reason why builds take so long. As we add more GWT tests we are going to test that LinkedIn record for the slowest build ever.

After this comment, I started looking into GWTTestSuite using Olivier Modica's blog entry as a guide. It was very easy to get things working in IDEA. However, when I'd run mvn test, I'd get the following error:

Error: java.lang.ClassCastException

No line numbers. No class information. Zilch. After comparing my project's pom.xml with the one from the default gwt-maven archetype, I noticed the default used JUnit 4.1, while I had the latest-and-supposedly-greatest JUnit 4.4. Reverting to JUnit 4.1 fixed the problem. Now Hudson takes 3:15 to execute the build instead of 15 minutes.

The reason for this blog post is this doesn't seem to be documented anywhere. Hopefully other developers will find this entry when googling for this issue.

Related to making GWT faster, I also added the following line to my Application.gwt.xml file:

<set-property name="user.agent" value="safari" />

This dropped the gwt:compile time from 1 minute to 25 seconds. As explained in the documentation, you can use the "user.agent" setting to only generate one JS file for your app instead of 4. The strange thing about adding this setting was I pretty much forgot about it since everything seemed to work fine on both Safari and Firefox. When I started testing things in IE6, I started seeing a lot of JavaScript errors. After debugging for an hour or so, I realized this setting was there, removed it, and everything started working great in all browsers.

Now if I could just figure out how to use safari-only for development, but remove the line when building the WAR. Suggestions welcome.

Posted in Java at Feb 27 2009, 11:58:12 AM MST 6 Comments

Enhancing your GWT Application with the UrlRewriteFilter

Last week, I spent some time trying to change the location of my cache/nocache HTML files in my GWT project. I started the project with the gwt-maven-plugin's archetype. The message I posted to the gwt-maven Google Group is below.

Rather than having my application's HTML file in src/main/java/com/mycompany/Application.html, I'd like to move it to src/main/webapp/index.html. I tried copying the HTML and adding the following to my index.html, but no dice:

<meta name="gwt:module" content="com.mycompany.Application"/>

Is this possible with the gwt-maven-plugin? I'd like to have my main HTML and CSS at the root of my application.

The good news is I figured out a solution using the UrlRewriteFilter that 1) allows hosted mode to work as usual and 2) allows your app to be served up from the root URL (/ instead of /com.company.Module/Application.html). Here's the urlrewrite.xml that makes it all possible.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCENGINE urlrewrite PUBLIC "-//tuckey.org//DTD UrlRewrite 3.0//EN"
        "http://tuckey.org/res/dtds/urlrewrite3.0.dtd">

<urlrewrite>
    <rule>
        <from>^/$</from>
        <to type="forward" last="true">/com.mycompany.app.Application/Application.html</to>
    </rule>
    <rule>
        <from>/index.html</from>
        <to type="forward" last="true">/com.mycompany.app.Application/Application.html</to>
    </rule>
    <-- This last rule is necessary for JS and CSS files -->
    <rule>
        <from>^/(.*)\.(.*)$</from>
        <to type="forward">/com.mycompany.app.Application/$1.$2</to>
    </rule>
</urlrewrite>

If you're using the gwt-maven plugin, this file goes in src/main/webapp/WEB-INF. In addition, you'll need to add the following to your web.xml.

    <filter>
        <filter-name>rewriteFilter</filter-name>
        <filter-class>org.tuckey.web.filters.urlrewrite.UrlRewriteFilter</filter-class>
    </filter>

    <filter-mapping>
        <filter-name>rewriteFilter</filter-name>
        <url-pattern>/*</url-pattern>
    </filter-mapping>

Finally, add the UrlRewriteFilter dependency in your pom.xml:

    <dependency>
        <groupId>org.tuckey</groupId>
        <artifactId>urlrewritefilter</artifactId>
        <version>3.1.0</version>
    </dependency>

Please let me know if you have any questions.

Update: Jeff posted an alternative configuration that allows you to eliminate the last rule in urlrewrite.xml, as well as use the beloved mvn jetty:run command. To use cleaner WAR packaging and the Jetty plugin, add the following to your pom.xml:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-war-plugin</artifactId>
    <configuration>
        <webappDirectory>
            ${project.build.directory}/${project.build.finalName}/com.mycompany.app.Application
        </webappDirectory>
    </configuration>
</plugin>
<plugin>
    <groupId>org.mortbay.jetty</groupId>
    <artifactId>maven-jetty-plugin</artifactId>
    <version>6.1.14</version>
    <configuration>
        <webAppConfig>
            <contextPath>/</contextPath>
            <baseResource implementation="org.mortbay.resource.ResourceCollection">
                <resourcesAsCSV>
                    ${basedir}/src/main/webapp,
                    ${project.build.directory}/${project.build.finalName}/com.mycompany.app.Application
                </resourcesAsCSV>
            </baseResource>
        </webAppConfig>
        <scanIntervalSeconds>3</scanIntervalSeconds>
        <scanTargets>
            <scanTarget>${basedir}/src/main/resources</scanTarget>
            <scanTarget>${basedir}/src/main/webapp/WEB-INF</scanTarget>
            <scanTarget>
                ${project.build.directory}/${project.build.finalName}/com.mycompany.app.Application
            </scanTarget>
        </scanTargets>
    </configuration>
</plugin>

Then you can trim your urlrewrite.xml down to:

<urlrewrite>
    <rule>
        <from>^/$</from>
        <to type="forward" last="true">/Application.html</to>
    </rule>
    <rule>
        <from>/index.html</from>
        <to type="forward" last="true">/Application.html</to>
    </rule>
</urlrewrite>

Of course, you could also change the welcome-file in your web.xml or use index.html and the <meta http-equiv="REFRESH"> option. Personally, I have so much affection for the UrlRewriteFilter that I like having it in my project. I'm sure I'll need it someday.

Thanks Jeff!

Posted in Java at Feb 23 2009, 05:02:29 PM MST 13 Comments

AppFuse Light converted to Maven modules, upgraded to Tapestry 5 and Stripes 1.5

This past week, I stayed up a couple of late nights to do some of the AppFuse Light work I wrote about in October. I converted all web frameworks to Maven modules, as well as made them inherit from the appfuse-web project. Below is what the new module structure looks like:

New AppFuse Light Modules

At this point, the project is ready to import into AppFuse's SVN project. Here's a list of other changes I made:

  • Modules now depend on AppFuse's backend and allow you to use Hibernate, JPA or iBATIS as the persistence framework. Implementations for Spring JDBC, OJB and JDO have been removed.
  • Upgraded to JWebUnit 2.1, which now uses HtmlUnit under the hood and has much better JavaScript support. It also has Selenium support, but I've yet to try it.
  • Ajaxified Body integrated into all frameworks. You can easily turn it off by modifying the global.js file.
  • Prototype and Scriptaculous loaded from Google's Ajax Libraries CDN.
  • Upgraded to Tapestry 5. Mad props to Serge Eby and his tapestry5-appfuse project for showing me how to do this. Serge became a committer on AppFuse recently, so hopefully we'll continue to see great things from the Tapestry 5 support. I really like the clean URLs and minimum configuration required in Tapestry 5. It's testing framework is nice too, but I believe it could be improved.
  • Upgraded to Stripes 1.5. This was easy and painless. I'm definitely a fan of Stripes and look forward to reading the Stripes book on my bookshelf.
  • Dropped support for: Struts 1.x, WebWork, Spring MVC + Velocity.

If you want to try any of these applications, you can create archetypes using the following commands:

svn co https://appfuse-light.dev.java.net/svn/appfuse-light/trunk appfuse-light
cd appfuse-light/preferred-web-framework
mvn archetype:create-from-project
cd target/generated-sources/archetype
mvn install
cd ~/dev
mvn archetype:generate # The new archetype should show up as an option

Next steps include figuring out a way to flatten the inherited dependencies and plugins so archetype:create-from-project can create truly standalone projects. Please let me know if you have any questions.

Posted in Java at Dec 20 2008, 06:42:03 PM MST 9 Comments

Free Maven Training in New Orleans on Election Day

Last week, I attended a talk by Jason van Zyl on Comprehensive Project Intelligence. This talk contained a good overview of some Maven goodies that Jason's company has been working on - namely better repository management (Nexus), better Eclipse support (m2eclipse) and improving an awesome CI server (we use Hudson heavily at LinkedIn).

Colorful New Orleans
Photo by Toshio

It looks like Jason is going to be doing a similar session in a full-day format at ApacheCon next week. The good news is it's free, the bad news is you have to be in New Orleans. Hopefully Jason and team will post their slides from this event and we can all learn how to use Maven more effectively.

In other Maven-related news, I hope to complete AppFuse 2.1 around the same time as Spring 3.0 (early next year). As part of that release, I hope to start using Nexus for AppFuse's repository as well as restructure its archetypes to allow using m2eclipse (with WTP) more effectively.

Posted in Java at Oct 29 2008, 09:55:45 PM MDT Add a Comment

AppFuse Light » AppFuse, Maven Archetypes and Shared Web Assets

Last night, I stayed up into the wee hours of the morning working on something I've been wanting to do for a long time. It wasn't until I was trouncing around the woods in Montana that I realized how easy it would be. The something I've wanted to do was to modify AppFuse Light to use AppFuse's core modules (service and dao). It only took me a few hours to make it happen and it inspired additional ideas.

I believe the major mistake we made in AppFuse 2.x was making it easy for user's to upgrade their applications. We currently use the maven-war-plugin and our own maven-warpath-plugin to make it possible to include AppFuse classes and assets in your project. You can easily start a new project w/o having a whole bunch of files in your project. The problem is, you can't easily use "mvn jetty:run" to work on your project. Of course, you can use "mvn appfuse:full-source" to solve this, but I'm starting to think more and more that "full-source" should be the default. This is what we did in 1.x and it seems to be the more natural pattern for folks using AppFuse.

That hard part about moving to "full-source" by default is coming up with a way to share common assets and classes among wars and war projects. Sure, I can copy all the shared images, css and js into each project - but that could become a maintenance nightmare. Subversion 1.5 with relative svn:externals might solve this, but it still seems kinda hacky. I don't want to use the maven-war-plugin because the overlay is kinda hokey and I think it's easier for users to understand when everything is in their project. AppFuse's current directory structure in SVN looks as follows. I've added indicators of what is in each directory.

AppFuse Web SVN

Rather than using AppFuse's current (manual) archetype-creation process, I'd like to move to a more automated creation process using the maven-archetype-plugin's create-from-project feature. I'd like to figure out a way where I can have the source code and assets from web/common included in each of the other web/* projects (both when using "jetty:run" and "archetype:create-from-project"). One idea I thought of is to make Jetty/Maven aware of multiple src/war directories for "jetty:run" and "package" and then somehow hook into the archetype plugin at creation time to pull in the shared resources. I don't know if something like this is possible. If you know of a good solution to this shared web assets issue, I'd love to hear about it.

Back to AppFuse Light. If I can figure out how to solve shared resources in web modules, I can use this in AppFuse Light to move to a modular SVN structure vs. its current "use Ant to create different combinations" setup. If a modular structure (like appfuse/web/*) is possible for AppFuse Light, I believe it makes sense to move its source into AppFuse's SVN repository. Below is how the directory structure might look after this move.

AppFuse Light » AppFuse

With this addition and "archetype:create-from-project", we should be able to create all the basic and light archetypes automatically. We'll probably still need a manual archetype-creation process for modular archetypes, but I'm OK with that.

The last thing I'm struggling with is figuring out the best way to create archetypes for something like AppFuse. In the past, we've used dependencies to allow users to inherit dependencies and their versions. This works, but it results in a lot of duplicate XML (in projects and archetypes) for developers. Last night, I tried using a parent project instead of dependencies and it seems to work much better. Not only do you inherit dependencies, but you also inherit plugins, profiles and properties. If you inherit, you can override, which is slick.

If you're an AppFuse user, how would you feel about having an AppFuse module as your project's parent? Would you prefer that, dependencies on AppFuse or full-source with no dependencies on AppFuse? Regardless of parent vs. dependencies, I think running "appfuse:full-source" should allow you to de-couple your project from AppFuse.

Posted in Java at Oct 29 2008, 02:18:59 AM MDT 7 Comments

Colorado Software Summit 2008 Wrapup

Snowman in Keystone Last week, I attended the Colorado Software Summit in Keystone and had a great time. Not only was the weather beautiful and the food delicious, the sessions I attended were awesome. If you ever get a chance to go to this conference, I highly recommend it. It's like being on vacation and learning with a bunch of old friends.

Yan Pujante also attended this conference and documents his experience, photography and presentations in Colorado Software Summit 2008.

Below is a list of my entries for all the sessions I attended.

For next year, I think the conference should shorten its sessions (from 90 to 60 minutes), invite more speakers and cut the price in half (to $999 per person). How do you think the Software Summit could be improved?

Posted in Java at Oct 28 2008, 11:03:23 PM MDT 2 Comments

Comprehensive Project Intelligence with Jason van Zyl

In this talk, Jason is going to talk about m2eclipse, Nexus, Hudson and Maven. On his Maven bullet-point, it says "The best is yet to come (and we'll fix a bunch of stuff)!"

m2eclipse
The m2eclipse plugin has improved greatly in the last 4 months - there's now 5 full-time developers working on it. If you use the m2eclipse plugin, you never have to leave the IDE for your Maven-related work. m2eclipse has a Configuration Framework that turns Maven's mumbo-jumbo (Jason's words, not mine) into Eclipse talk. The m2eclipse+configuration framework has integration for WTP, JDT, AJDT and they're working on one for Flex. Below is a screenshot of how m2eclipse helps developers stay away from using command-line Maven.

m2eclipse Configuration Framework

Now Jason is showing a demo of m2eclipse and creating a new Maven project from existing archetypes. It looks like m2eclipse uses "Nexus Indexer" as its Catalog. Presumably this is a Sonatype-hosted service. The Nexus Indexer contains an of Maven Central and is very fast. It's dynamically updated as new things are deployed to Maven Central.

If you use m2eclipse and open a pom.xml, you'll get a visual view rather than an XML view. This UI has tabs for Overview, Dependencies, Repositories, Build, Plugins, Reporting, Profiles, Team, a Dependency Hierarchy and Dependency Graph. You can easily add new dependencies and it finds things quickly because it's using the Nexus index. In addition to visually adding dependencies, you can modify the raw XML and get things like groupId and version code-completion.

Once you have your dependencies listed in a "Maven Dependencies" container to you can "Materialize Project" to create a project from the binary dependency. You'll get the source as a new project in your workspace as well as having your binary dependency turned into a source dependency.

You can easily create a run configuration that runs certain goals, allows you to activate profiles and uses an embedded version of Maven or an external installation. I asked Jason if the Dependency Hierarchy had a right-click -> exclude feature and he said it doesn't exist yet, but it will in the release after next. For now, the pom editor is just eye candy and doesn't have actions.

For Maven Plugins, m2eclipse has workspace resolution so you can develop a plugin and use it in a project at the same time w/o having to install the plugin over and over.

Sonatype has created a Project Materializer Plugin that allows a team lead to create a project for developers. It allows you to create a welcome page that has links, cheat sheets, News and Updates and Tasks for the developer. It also materializes Eclipse projects in the background. Cheat Sheets are a series of tasks that can be run to show developers how to do things.

Another big feature in m2eclipse is nested project support. It only works in Eclipse 3.4 though.

Nexus
Nexus is a repository manager that allows you to keep the cruft from the outside world out of your system. It's primarily for Repository and Configuration Management. It has fine-grained security for authentication and authorization. One nice feature of its security system is you can prevent certain users from seeing source JARs. It also has virtual repositories (a savior for OSGi lovers). UI is written in Ext JS and acts as a simple REST client for Nexus. It has a full REST API using RESTlet.

A repository manager allows you to protect yourself from the Open Source Ghetto. The OS Maven Ghetto has bad POMs, repositories in POMs, mixed snapshot and release dependencies and screwed up metadata. Not only does it offer protection, but it allows you to aggregate repositories and publish your internal artifacts to it. It also allows you to schedule tasks that clean out snapshots so your repositories don't grow out of control.

Typically people deal with OSGi runtimes manually. OSGi can dynamically update dependencies that you drop into your bundle repository. However, many folks maintain their OSGi runtime and bundle repository locally. Some people are trying to get an OSGi runtime to resolve against a P2 repository. P2 is what Eclipse uses for their repository management. Nexus has the ability to lock down the versions that are available to an OSGi runtime. Furthermore, you can use Nexus to manage the versions that get deployed to all your servers. This makes it a lot easier for QA and Production to manage versions of your artifacts. OSGi is great for modularity and solving classpath issues, but it does have issues with versions and how its ranges work.

You can see Nexus in action at http://repository.sonatype.org. It can be configured entirely through the UI, an XML file or through the REST API. RSS feeds exist for configuration and repository updates.

Nexus is free and open source with a GPL license. The next version (1.2) will contain a Plugin API to allow extensions. All of Sonatype's enhancements for its commercial version will be written as plugins. A matrix of what's available in the open source version vs. commercial version should be published sometime next week.

Hudson
Jason believes that Hudson is the future of continuous integration, on-demand results and release management. They're writing all their extension points in Hudson as Maven plugins and Plexus components (with the work they've done, using Spring components should also be possible). Other enhancements they've made to Hudson:

  • Integration of JSecurity
  • Implementing a similar REST layer as Nexus and creating a UI using Ext JS
  • Automatic installation of external Maven installations
  • Drools Workflow Integration

They've also enhanced Hudson so it can easily test/publish Maven projects without using the free-form project template. Hudson works well for doing Eclipse headless builds for Eclipse plugins. If you need to test against multiple databases, multiple OS's, it does support a grid-based system that's easy to setup. Hudson does have web services integration that allows you to kick off builds from within Eclipse. Sonatype uses Hudson to run all their nightly builds of Maven.

Maven - the best is yet to come
The three big things coming in the next version of Maven are:

  • Refactored Project Builder: includes a spec for building a pom, domain-specific parsers (attribute-based XML, Groovy and Ruby) and mixins.
  • Mercury: a new repository and transport layer. Developed by the Jetty people and is super fast (async client with connection pooling and parallelization). Has atomic downloads and deployments (with Nexus), full PGP support and a WebDAV client built-in.
  • Maven Embedder: re-written to actually work.

Overall, a good talk with lots of demos. I'm definitely looking forward to Maven improvements in the future.

Posted in Java at Oct 21 2008, 02:35:13 PM MDT 7 Comments

Maven Plugin for Running Integration Tests against Multiple Containers

Don Brown has created a pretty cool Maven plugin that allows you to run integration tests on multiple containers in one go. I learned about it on the Struts 2 Dev List:

I've started adding functional tests to Struts 2 by adding a few to the REST showcase application, running against Tomcat 5.x, Jetty 6.x, JBoss 4.2.x, and Resin 3.x. The magic happens through a new Maven 2 plugin I developed called maven-itblast-plugin, which enables multiple integration test runs against multiple containers in one go. For more info, see http://github.com/mrdon/maven-itblast-plugin/wikis/home.

You might notice that this plugin is hosted on GitHub. If you want to learn how to use GitHub with Maven, you might want to see Don's entry on Maven-enabled project hosting with GitHub. Thanks Don - good stuff.

Update: Don has posted more information about this plugin on his blog.

Posted in Java at Jul 31 2008, 05:39:23 PM MDT Add a Comment

Issues with AntRun Plugin and Maven

I started seeing the following error today when using Maven and the AntRun Plugin.

[INFO] [antrun:run {execution: default}]
[INFO] Executing tasks
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Error executing ant tasks
	
Embedded error: java.lang.IllegalAccessError: tried to access method 
org.apache.tools.ant.launch.Locator.decodeUri(Ljava/lang/String;)Ljava/lang/String; 
from class org.apache.tools.ant.AntClassLoader

Searching the internet provided no results, so I was pretty stumped - especially since this error didn't happen on my MacBook Pro. It happened on AppFuse's Bamboo server (Linux), but not locally. Luckily, I was able to reproduce it on my Windows box and discovered the solution: upgrade to a newer version of Maven. I was using 2.0.6/2.0.7 and upgrading to 2.0.9 fixed the problem.

BTW, when is the Ant project going to release a new version of Ant? The current 1.7.0 version doesn't support spaces in path names, which seems like a pretty big issue to me (especially for Windows users).

Posted in Java at May 10 2008, 06:23:26 PM MDT 6 Comments