Raible Designs

Comparing Open Source Application Servers

With all the recent hubbub about GlassFish, I decided to do a quick performance test this morning. I downloaded all the most recent versions of the various open source application servers, deployed AppFuse 1.9.3 (Struts version) on them, and ran "ant test-canoo" to see if any of them were faster than the other. This was by no means a scientific, isolated test. It also didn't take into account any performance tuning you should do on these servers, I just used the out-of-the-box settings.

I ran these tests on my MacBook Pro (2.16 GHz Intel Core Duo, 2 GB DDR2 SDRAM) with my JAVA_OPTS set to:

-Xms768M -Xmx768M -XX:MaxPermSize=512m -Djava.awt.headless=true

When typing "java -version" at the command line, I got:

java version "1.5.0_06"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-103)
Java HotSpot(TM) Client VM (build 1.5.0_06-57, mixed mode, sharing)

Servers tested (in no particular order):

• JBoss 4.0.4
• GlassFish B48
• JOnAS 4.7.6
• Resin 3.0.21
• Geronimo 1.1

I'm pleased to note that all servers allowed me to deploy appfuse.war without using a console or command-line tool. They all support dropping the WAR in some sort of auto-deploy directory. Very cool! Secondly, I was able to successfully deploy AppFuse on all of them with no changes to AppFuse nor the server. Quite impressive.

My test consisted of the following:

• Copying appfuse.war into the appropriate directory
• Starting the server
• Running "ant test-canoo" from my $APPFUSE_HOME directory once
• Running "ant test-canoo" 3 times, recording the numbers for each run

Here's what I found:

Since I know you're going to ask about Jetty and Tomcat (the two main servlet-only containers), I ran the numbers on those too. First off, I tried Jetty 6 RC0. No dice - I got the following error when trying to start the server.

java.lang.IllegalStateException: Cannot initialize context because there is already a root application 
context present - check whether you have multiple ContextLoader* definitions in your web.xml!
        at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:173)

Since AppFuse deploys on all the above app servers, as well as Jetty 5.1.x, I'll chalk this up to a bug in Jetty 6. I used Jetty 5.1.11 for this test because I already had it installed on my machine.

I don't know that these numbers mean anything, but it was a fun experiment. For those of you who think these numbers might mean something, here's the rankings:

1. Tomcat 5.5.17
2. Resin 3.0.21
3. JBoss 4.0.4
4. Jetty 5.1.11/GlassFish B48 (tie)
5. Geronimo 1.1
6. JOnAS 4.7.6

Of course, the better test would be hammering each server with 1000 concurrent users (or a number higher than that) and comparing how each server holds up.