Raible Designs

[OSCON 2008] Even Faster Web Sites by Steve Souders

Steve works at Google on web performance and open source initiatives. To begin his talk, Steve is running YSlow in "autorun" mode. This runs YSlow Performance tests on the top 100 sites according to Alexa. Before Google, Steve was at Yahoo for 7 years. You can download Steve's slides from his site (don't ask me where).

iGoogle with an empty cache: 9% of the time is spent getting the HTML document. The % of time of what a webserver does is a pretty small percentage of the overall picture. If the cache is primed, the time goes up to 17% of the time.

80-90% of the end-user response time is spent on the frontend. Start there.

There's a greater potential of improvement on the frontend. If you improve the backend performance by 50%, chances are the end-user only sees a 5% improvement.

The 14 Rules are encapsulated in the YSlow plugin. At OSCON last year, Yahoo released YSlow. 500,000 downloads since it was released. Following the release of YSlow, Steve wrote High Performance Web Sites.

High Performance Web Sites, Vol 2. The 3 most important rules are:

• Split the initial payload
• Load scripts without blocking
• Don't scatter inline scripts

Why focus on JavaScript? A lot of the top sites use JavaScript. For example, up until a few weeks ago, Facebook served up 1MB of JavaScript, uncompressed. Scripts block parallel downloads and page rendering. To see it in action, go to http://stevesouders.com/cuzillion/?ex=10008.

Any content below a <script> is blocked from rendering - even if it's already cached in the browser. Cuzillion is an open source project that Steve is releasing that allows you to add components to a page and test their performance.

Split your JavaScript between what's needed to render the page and everything else. Load "everything else" after the page is rendered. To do this, you can use Firebug to do it manually, or you can use Doloto from Microsoft to automate the splitting of your files.

MSN.com solves the script blocking problem by using JS DOM to allow for parallel downloading. There's 6 techniques for doing this:

• XHR Eval (must have same domain as page)
• XHR Injection (same domain)
• Script in iFrame (same domain)
• Script DOM Element (domains can differ)
• Script Defer (only supported in IE, domains can differ)
• document.write (not recommended, parallelization only works in IE)

How do these techniques cause "browser busy" indicators? XHR Eval and Injection don't trigger any indicators. You need to choose when you want to show busy indicators. It's good to show them when you want to show your users that something is processing (but not for lazy-loading JavaScript that's not required for load). For the different techniques, most don't ensure order of parsing.

Based on 3 factors, Steve can tell you which technique is best to use. These three factors are 1) the URL of the page and script 2) if you want busy indicators and 3) if you care about order. Steve thinks it would be awesome if web frameworks could support this to write out JavaScript appropriately for the developer's input.

Long executing inline scripts block rendering and downloads. If you know you're going to have scripts like this, you can solve it with a couple workarounds:

• Initiate execution with setTimeout(>250 for Firefox)
• Move JavaScript to enternal script with advanced downloading techniques
• Use defer attribute for IE

In Firefox 2, stylesheets block parallel downloads just like scripts. IE doesn't. However, IE will block when you have a stylesheet followed by an inline script. To solve, it's best to move line scripts above stylesheets or below other resources. use <link>, not @import.

Takeaways: focus on the frontend, run YSlow, focus on JavaScript (split initial payload, load scripts w/o blocking, don't scatter inline scripts).

Three Announcements:

• HTTPWatch for Firefox (not public yet, not free).
• Firebug Light 1.2 (can be used with IE or Opera).
• Mozilla is Firebuggin' (Mozilla is dedicating 3 people to work on Firebug).

[OSCON 2008] The State of Lightning Talks

From this session's detail page:

As a repeat of the last 2 years, 15 open source project leaders will be given five minute lightning talk slots to bring the audience up to date on their projects.

This year’s speakers and projects include: Brian Aker: Memcached, Glynn Foster: OpenSolaris, OSI: Danese Cooper, MySQL: Monty Widenius, PostgreSQL: Bruce Momjian, GNOME: Dave Neary, Gentoo: Donnie Berkholz, OpenOffice.org: Louis Suarez-Potts, Jabber: Peter Saint-Andre, Mozdev: Brian King, OpenID: Scott Kveton, Open Scrum: James Dixon, and Cliff Schmidt for the Talking Book Project. A couple more projects may be added later.

Each speaker has exactly 5 minutes, and we use various “fun” tools to make sure they stick to their time. The session usually ends with a “zinger” presentation worth staying for.

Unlike the description above, apparently there's 17 projects instead of 15. This session is 95 minutes long (spans 2 normal sessions). I figure this will be the most challenging event for me to blog, so here we go!

OSI (Open Source Initiative), Danese Cooper
Danese has been on the board of OSI since 2001 and will be on the board for another 3 years. This year, they've been asking themselves "What are the ways to make OSI more effective?" The best answer they've come up with is that they should run OSI more like an open source project (with lots of transparency). Minutes are posted publicly on website and they have a public blog. They're also transferring their archives to a searchable system. They'll be using Trac to do more objective tracking of all the issues that've been brought to their attention.

They're also going to establish (formal) membership. They're going to use Apache as a model and allow Charter, Individual, Organizational and have an Annual Members Meeting. ASF currently has 300 members and these folks are in charge of making decisions for the foundation. There's a good chance you'll recognize all the Charter Members of OSI. The names will be announced in the next couple months (I believe). There will be opportunities to be individual or organizational members.

There's new board members this year with an international focus: Nnenna (Africa), Harshad Gune ...honk... out of time.

Open Scrum, James Dixon
James is the founder and Chief Geek at Pentaho. Pentaho recently took their products open source and they're releasing Open Scrum as an agile methodology for managing open source projects. Open Source is not a methodology, it is a set of principles.

Some agile principles are used in open source, but some are not. For instance, trusting developers and face-to-face contact can be difficult with open source projects. Simplicity sometimes comes out of open source projects, but usually only for projects with a large developer base. There's a 45% good fit between Agile and Open Source.

Open Scrum is based on Scrum, but geared towards open source projects. It's simple and flexible and template driven. It scales from single-geek, part-time, single-projects to massively multi-geek full-on and multi-layered undertakings.

MySQL, Monty Widenius
Monty is the Chief Fonder of MySQL and he's talking about what's new in MySQL. There's been no new releases for users, but internally they've been doing lots of bug fixes in 5.1. He expects to have a reasonable GA of 5.1 in 3-5 months. 5.0 will include online backup and optimizer changes for faster joins and sub queries. 5.1 has an InnoDB Plugin.

MySQL was acquired by Sun for $1 billion and 6 months later, life is good. MySQL Backup was announced to be crippleware by MySQL management, but later changed to be open source.

Switched to use Bazaar and Launchpad to increase community participation. Started public bug hunt for MySQL 5.1. Started using case competition for new features in 5.1. There's also a new MySQL forge. Please help MySQL get better by filing bugs!

Maria engine 1.0 and 1.5; now crashsafe and multi-versioned. Transactional part to follow soon.

OpenID, Scott Kveton
Scott is on the board of the OpenID Foundation and his son is on stage with him (with a Twitter shirt on). In terms of adoption, they have a hockey stick graph with the big upturn starting in February 2007. OpenID is only 3 years old and there's over 20,000 sites that now support it. There's about ~500 million OpenIDs (including every AOL, Yahoo, MySpace and LiveJournal user).

OpenID 2.0 was released as final 12/7/2007. What's next: usability (currently sucks), security (PAPE), integration with other protocols, more than just providing, and possibly OpenID 3.0. One of the biggest problems with usability is people just can't remember URLs, so they're trying to figure out a URL -> e-mail mapping.

OpenID Foundation has added 5 corporate board seats and has 7 community board seats and elections are in August. You can get code at openidenabled.com and you can get involved at openid.net.

Gentoo, Donnie Berkholz
Donnie is one of 7 people that's on the council that tries to make sure Gentoo is going in a good direction. In the last year, they've been struggling with community and releasing. Gentoo is just getting old enough where people are quitting and new developers are coming on board. In 2005, they had a lot of voters for their council. 2006 and 2007 were bad years from a community participation standpoint. This year, they've fixed "poisonous people" and participating is back up.

Rather than voting for single candidates, they use a "rank all the candidates" voting system. Community is very important and they've recently kicked out 3 people that didn't think community was important. They released 2008.0 a few weeks ago. It was supposed to be released in February. When they released, they made the front page of Digg, which made Donnie very happy. Gentoo news announcements were way down in 2007, but now they're back up.

Recommended books for community building: Good to Great, The No Asshole Rule, Getting to Yes.

PostgreSQL, Bruce Momjian
Bruce is a PostgreSQL core developer and works for EnterpriseDB. The future for PostgreSQL is going to be a lot different than the past. PostgreSQL's path to the future isn't a straight line. Rather, it's a meandering path like a road up a mountain with switchbacks. People would like a catapult to put features into to get things in, but it's not that easy. It's difficult to improve software that's been around for 20 years.

Companies have very clearly defined goals that usually revolve around money. With open source, it's a lot different. Goals are usually for a feature set, for adoption of open source or for the challenge and fun of it.

PostgreSQL 8.0 was released in January 2005 and had a lot of enterprise features: Win32, Savepoints, Point-In-Time Recovery, Tablespaces. The latest release (8.3 in February) was much different and much more exotic features. In the future, they're going to expand on running queries on multiple CPUs and maintaining their leadership in the open source database space.

Bazaar, Mark Shuttleworth
Mark's interest in distributed version control predates his interest in Ubuntu. If we can elevate our process of version control, we can elevate the entire open source community. Bazaar is a distributed version control system that's developed in Python. It focuses relentlessly on a couple of key values:

• Cross-platform: Linux, Windows, Mac OS X
• Performance: sub-second status on > 10,000 files
• Being adaptive to the way that people want to work (can use it like Subversion if you want or you can have distributed branches or have a team checkout)
• Extensibility and embeddable - has a plugin system
• Python Hackable - people can easily modify

Projects that use Bazaar: GNU, MySQL, Twisted and Ubuntu.

Memcached, Alan "Dormando"
Memcached is a very simple project with a mailing list with very simple people on it that run very simple websites. They haven't had a release in 6 months. Memcached was written by a guy named Fitz that wrote it and left. They've only recently created a developer community that's all over the globe. Alan didn't bring any slides, but he did release 1.2.6 RC1 as he was standing on stage. They also hope to release 1.3.1 this week, but there's not a lot of documentation.

People fork Memcached a LOT, mostly because they need a new backend. They've added a new storage engine that's pluggable to hopefully fix the forking problem. Also, their new binary protocol should make it a lot faster for high traffic web sites.

The Talking Bridge Project, Cliff Schmidt
While the network at OSCON has been slow, the folks that live in rural areas around the world have an incredibly slow means of getting information. Their network is often roads that people travel on to communicate with each other.

They've created a Talking Book device that is an MP3 player that costs $5-10. They're also creating kiosks that people in rural areas can interact with to download new information (i.e. podcasts and other educational content). It also has a microphone and can be found at literacybridge.org. They're creating open source projects to help build the device and content around it. They'd love to have open source developers join their projects and help them make access to information easier for rural societies.

Open Source Lab at Oregon State University, Lance Albertson
Lance is a Kansas native and grew up on a farm. He's a Gentoo developer and joined OSL a year ago. OSL was created in 2003 to utilize "extra" unused bandwidth. They provide managed and co-located hosting for open source software projects. They also provide on-the-job learning opportunities for students in system administration and development.

In addition to hosting, they have a development and outreach side. They have 12 student employees that are given developers/sys admins are given real world practical opportunities. Their work is seen and used by many people and students are given extraordinary amounts of responsibility. They're working on a couple external OSS projects: Oregon Virtual School District and One Laptop Per Child.

Outreach wise, they have an Open Source Education Lab and they've been doing Drupal training for OSU employees. They've also helping organize GOSCON, which is an open source conference for governments.

They currently offer hosting for over 70 OSS projects. Services include, email relays, databases, web hosting, file mirroring and code repositories. Notable clients: ASF, Drupal, kernel.org (master), OpenOffice, Freenode, Gentoo Linux & Debian, phpBB and many others. Datacenter currently has 50 racks; they'll be moving to 75 in the near future.

OpenOffice
They've had 150 million downloads and believe there's around 1 million users. There's 350 committers and most of them work from Sun. Presently, their biggest focus is Desktop. In the future, they'll focus on the Web. The idea is that people will want to use OpenOffice from their mobile devices. ODF is a format that can be expressed by any application. OO 3.0 is about extensions, not bloat. They're adding linkage to a free PIM (Outlook replacement). They're also adding better interoperatbility with other suites and formats (e.g., Microsoft Office 2007). They're adding productivity tools and adding toolkits to make things easier for developers. In the next year, they expect to go way beyond 150 million downloads. They also expect to expand greatly internationally.

Mozdev, Brian King
Brian has been in the Mozilla Community since 1999. Mozdev.org is a community site that provides free hosting for Mozilla applications and extensions since 2000. Established as a non-profit organization in 2004. It has over 250 active projects today and there's over 500 developers discussing issues on the [email protected] list.

Services provided: source code hosting, bug tracking (Bugzilla), communication tools (lists, newsgroups, blogs, forums), file management tools, project tagging, wiki, statistics, permissions system and public planning.

They've experienced some grown pains in the last few years, but they're mostly solved now. They've added developer resources to help develop their site, but they need more. Up until Firefox, they were the #1 place to go for Mozilla projects. They're still working on letting people know that they're around and have projects that are interested. They're looking for new avenues for funding.

Moving forward: they'll be add more items to roadmap, making it easier for less technical folks to host and create extensions, solving the funding issue, and developing deeper relationship with Mozilla (MoFo and MoCo). Quite simply, they want to continue with their goal of supporting developers in order to proliferate Mozilla technologies.

Glen Foster, OpenSolaris
Hey - I had a beer with this guy last night at Kells! Rather than focusing on the older Solaris users, they want to move to focus on getting kids to learn and use it. A couple of months ago, they released 200805 and reduced 5 CDs down to a single live CD. The packaging system is now network based. They changes the default shell to bash and changed to GNU tools as they default as well.

They introduced a new site - opensolaris.com. Since Sun released the source code 3 years ago, they've had a long evolution of getting the source out the door. They were using Teamware and doing much of the development behind the door. Early next month, they're moving to Mercurial and they'll be moving all the source outside the firewall. They'll be using Bugzilla as their bug tracking system.

Similar to Gentoo, they've had quite a few people in their community that don't like change. They're working on fixing this and have had good progress. Several years ago, they created a governance system before they had a community. This was a mistake and their new community-driven governance system should work much better.

GNOME, Dave Neary
GNOME is a desktop environment and set of utilities. It's the default on Ubuntu and Fedora. It's the face of Linux on the Desktop. It's also a platform for development. They have a vibrant ISV community that includes Adobe, VMWare and IBM. In addition to desktop, GNOME is also a mobile platform. They also act as a big tent that provides an infrastructure for projects that are part of the GNOME ecosystem.

The most important thing to Dave is the shared vision of universal access to anyone and everyone. Very important things to the GNOME project: usability, internationalization and localization, accessibility (Sun and IBM have done a lot of work). Accessibility is something they've really focused on in the last year. They've created an Outreach Project for Accessibility and have gotten funding from Mozilla, Canonical and Google. Mozilla has done a lot to fund them and help provide accessibility for Rich Internet Applications. To see if a website is accessible, try it in lynx.

Subversion, John Mark Walker
John is the Community Manager for openCollabNet and is not a Subversion developer. Subversion 1.4 was released in September 2006. There was much anticipating for 1.5 and was released last month. The people rejoiced.

What's new in 1.5? Merge tracking: easier to merge changes from trunk to branch and vise versa. You can also do sparse checkouts (replaces -N). Interactive conflict resolution so you can do it from the command line client. It also add changelist support. svn:externals is no longer required to be absolute (sweet!) and you can add peg revisions in your URLs. FSFS repos never change a rev after written to disk. Clients many now perform chained copy/move operations locally. It also supports moving multiple sources for copy and move. Client operations are now significantly more responsive to canceling operations. "resolve" subcommand replaces "resolved" (deprecated). Delete (remove) now takes a --keep-local option.

Now it's time for the fun stuff...

Ken Drachnik had to leave for an emergency and is unable to give his talk on GlassFish. An audience member (Carl Fogel) volunteers to do his presentation. It's very funny, especially since he's not a Java Developer and has never heard of GlassFish. The only thing I got out of this presentation (besides lots of laughs) is that v3 will be released in June 2009.

Drizzle, Brian Aker
Stored procedures, triggers, prepared statements and many other things have been discarded. "One CPU to rule them, and in the query cache, bind them." If you have a query cache turned on in our database, turn it off or delete it.

Drizzle is currently ~420K. The Master Plan is to rethink everything and not assume everything was bad. Also, they can reuse many other libraries w/o writing them from scratch. They don't have to ship every library because many are present on operating systems. In with the new: C99, Posix, package-lib. They're moving to a MicroKernal design and moving code to the edge. No new features will be in core. To add interfaces you have to remove code so there's an equal amount of code.

Multi-Core support: no new locks, remove old locks. Think today (burn memory, messages scales). They can leverage Sun because they can build machines with more cores than anyone else.

Field Types: Less primitives - now there's just one "blob". Removed the 3 byte int and add UUID/IPV4(6). They dumped ACL because we authenticate in clouds. KISS.

You can get involved with Drizzle at http://launchpad.net/drizzle (bzr branch lp:drizzle) and http://launchpad.net/drizzle-discuss (mailing list).

Is it possible to replace the syntax parser in Eclipse or IDEA's JSP Plugin?

At LinkedIn, we have our own JSP Compiler. Our version of JSP is more like FreeMarker than JSP since it solves many of the deficiencies of JSP. Since we allow a different syntax than standard JSP (more powerful EL, new tags for looping, loading from classpath), we (like FreeMarker) don't get much love from IDEs.

We don't get much in the way of syntax-highlighting or code completion. However, since we use JavaCC/JJTree for parsing, I'm wondering if Eclipse or IDEA (or even NetBeans) allows replacing the default syntax definition with a new one.

Has anyone extended one of these IDEs to enhance its JSP syntax highlighting and compilation? If so, I'd love to hear about it. If not, it's likely we'll be doing it in the near future.

Should we change AppFuse to be Struts 2-specific?

Dusty recently posted an interesting idea to the AppFuse developers mailing list:

After thinking/coding/reading for a while I think the more interesting task is: Retool AppFuse to be one or more Struts2 plugins based on various higher level app patterns. (AppFuse Facebook, AppFuse Employee DB, AppFuse Blog, AppFuse Basic LDAP, AppFuse Basic Crowd).

This all comes from the fact, that I have been wanting to refactor the AppFuse web layer for Struts. One of the interesting aspects of AppFuse is that it works pretty much the same across all its web frameworks. It does so with some lowest common denominator abstractions that can be ported and look and work the same across frameworks. I have picked my tool(s): Struts 2 and Ruby On Rails when I want to pretend I am young again. I know Spring MVC, JSF, etc. but I have no desire to build significant apps on those platforms. It's not because they suck and Struts2 rules, it is because I know Struts 2 the best, I am most efficient there and it provides everything I need to build great webapps (Let's not devolve to a framework debate). So, I would rather have a more Struts 2-specific web stack, that really leverages conventions born and raised there. The nice thing about the Struts 2 web stack is that it is complemented nicely by AppFuse's data/service layer, since unlike Grails or Rails, Struts 2 has no data or service layer. [Read More »]

Seems like a good idea to me. What do you think?

Someday I'd like to come up with a "compatibility test" that allows others to improve upon the ideas in AppFuse and develop their stacks independently. A suite of Selenium tests that require extensionless URLs might be a good start.

AppFuse 2.0.2 Released

The AppFuse Team is pleased to announce the release of AppFuse 2.0.2. This release includes upgrades to Spring Security 2.0, jMock 2.4, the ability to customize code generation templates and many bug fixes.

For information on upgrading from 2.0.1, see the Release Notes or changelog. AppFuse 2.0.2 is available as a Maven archetype. For information on creating a new project using AppFuse, please see the QuickStart Guide or the demos and videos.

To learn more about AppFuse, please read Ryan Withers' Igniting your applications with AppFuse.

The 2.0 series of AppFuse has a minimum requirement of the following specification versions:

• Java Servlet 2.4 and JSP 2.0 (2.1 for JSF)
• Java 5+

If you've used AppFuse 1.x, but not 2.x, you'll want to read the FAQ. Join the user mailing list if you have any questions.

Thanks to everyone for their help contributing code, writing documentation, posting to the mailing lists, and logging issues.

Please post any issues you have with this release to the mailing list.

Issues with AntRun Plugin and Maven

I started seeing the following error today when using Maven and the AntRun Plugin.

[INFO] [antrun:run {execution: default}]
[INFO] Executing tasks
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Error executing ant tasks
	
Embedded error: java.lang.IllegalAccessError: tried to access method 
org.apache.tools.ant.launch.Locator.decodeUri(Ljava/lang/String;)Ljava/lang/String; 
from class org.apache.tools.ant.AntClassLoader

Searching the internet provided no results, so I was pretty stumped - especially since this error didn't happen on my MacBook Pro. It happened on AppFuse's Bamboo server (Linux), but not locally. Luckily, I was able to reproduce it on my Windows box and discovered the solution: upgrade to a newer version of Maven. I was using 2.0.6/2.0.7 and upgrading to 2.0.9 fixed the problem.

BTW, when is the Ant project going to release a new version of Ant? The current 1.7.0 version doesn't support spaces in path names, which seems like a pretty big issue to me (especially for Windows users).

Running Spring MVC Web Applications in OSGi

For the past couple of weeks, I've been developing a web application that deploys into an OSGi container (Equinox) and uses Spring DM's Spring MVC support. The first thing I discovered was that Spring MVC's annotations weren't supported in the M1 release. This was apparently caused by a bug in Spring 2.5.3 and not Spring DM. Since Spring DM 1.1.0 M2 was released with Spring 2.5.4 today, I believe this is fixed now.

The story below is about my experience getting a Spring MVC application up and running in Equinox 3.2.2, Jetty 6.1.9 and Spring DM 1.1.0 M2 SNAPSHOT (from last week). If you want to read more about why Spring MVC + OSGi is cool, see Costin Leau's Web Applications and OSGi article.

To get a simple "Hello World" Spring MVC application working in OSGi is pretty easy. The hard part is setting up a container with all the Spring and Jetty bundles installed and started. I imagine SSAP might solve this. Luckily for me, this was done by another member of my team.

After you've done this, it's simply a matter of creating a MANIFEST.MF for your WAR that contains the proper information for OSGi to recognize. Below is the one that I used when I first tried to get my application working.

Manifest-Version: 1
Bundle-ManifestVersion: 2
Spring-DM-Version: 1.1.0-m2-SNAPSHOT
Spring-Version: 2.5.2
Bundle-Name: Simple OSGi War
Bundle-SymbolicName: myapp
Bundle-Classpath: .,WEB-INF/classes,WEB-INF/lib/freemarker-2.3.12.jar,
 WEB-INF/lib/sitemesh-2.3.jar,WEB-INF/lib/urlrewritefilter-3.0.4.jar,
 WEB-INF/lib/spring-beans-2.5.2.jar,WEB-INF/lib/spring-context-2.5.2.jar,
 WEB-INF/lib/spring-context-support-2.5.2.jar,WEB-INF/lib/spring-core-2.5.2.jar,
 WEB-INF/lib/spring-web-2.5.2.jar,WEB-INF/lib/spring-webmvc-2.5.2.jar 
Import-Package: javax.servlet,javax.servlet.http,javax.servlet.resources,javax.swing.tree,
 javax.naming,org.w3c.dom,org.apache.commons.logging,javax.xml.parsers;resolution:=optional,
 org.xml.sax;resolution:=optional,org.xml.sax.helpers;resolution:=optional

Ideally, you could generate this MANIFEST.MF using the maven-bundle-plugin. However, it doesn't support WARs in its 1.4.0 release.

You can see this is an application that uses Spring MVC, FreeMarker, SiteMesh and the URLRewriteFilter. You should be able to download it, unzip it, run "mvn package" and install it into Equinox using "install file://<path to war>".

That's all fine and dandy, but doesn't give you any benefits of OSGi. This setup works great until you try to import OSGi services using a context file with an <osgi:reference> element. After adding such a reference, it's likely you'll get the following error:

SEVERE: Context initialization failed
org.springframework.beans.factory.parsing.BeanDefinitionParsingException:
Configuration problem: Unable to locate Spring NamespaceHandler for
XML schema namespace [http://www.springframework.org/schema/osgi]

To fix this, add the following to your web.xml (if you're using ContextLoaderListener, as an <init-parameter> on DispatcherServlet if you're not):

  <context-param>
    <param-name>contextClass</param-name>
    <param-value>org.springframework.osgi.web.context.support.OsgiBundleXmlWebApplicationContext</param-value>
  </context-param>

After doing this, you might get the following error on startup:

SEVERE: Context initialization failed
org.springframework.context.ApplicationContextException: Custom
context class [org.springframework.osgi.web.context.support.OsgiBundleXmlWebApplicationContext]
is not of type [org.springframework.web.context.ConfigurableWebApplicationContext] 

To fix this, I change from referencing the Spring JARs in WEB-INF/lib to importing the packages for Spring (which were already installed in my Equinox container).

Bundle-Classpath: .,WEB-INF/classes,WEB-INF/lib/freemarker-2.3.12.jar,
 WEB-INF/lib/sitemesh-2.3.jar,WEB-INF/lib/urlrewritefilter-3.0.4.jar
Import-Package:
javax.servlet,javax.servlet.http,javax.servlet.resources,javax.swing.tree,
 javax.naming,org.w3c.dom,org.apache.commons.logging,javax.xml.parsers;resolution:=optional,
 org.xml.sax;resolution:=optional,org.xml.sax.helpers;resolution:=optional,
 org.springframework.osgi.web.context.support,
 org.springframework.context.support,
 org.springframework.web.context,
 org.springframework.web.context.support,
 org.springframework.web.servlet,
 org.springframework.web.servlet.mvc,
 org.springframework.web.servlet.mvc.support,
 org.springframework.web.servlet.view,
 org.springframework.ui,
 org.springframework.web.servlet.view.freemarker 

After rebuilding my WAR and reloading the bundle in Equinox, I was confronted with the following error message:

SEVERE: Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error
creating bean with name 'freemarkerConfig' defined in ServletContext
resource [/WEB-INF/myapp-servlet.xml]: Instantiation of bean failed;
nested exception is java.lang.NoClassDefFoundError:
freemarker/cache/TemplateLoader
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:851) 

As far as I can tell, this is because the version of Spring MVC installed in Equinox cannot resolve the FreeMarker JAR in my WEB-INF/lib directory.

To prove I wasn't going insane, I commented out my "freemarkerConfig" and "viewResolver" beans in myapp-servlet.xml and changed to a regular ol' InternalResourceViewResolver:

<bean id="viewResolver" class="org.springframework.web.servlet.view.InternalResourceViewResolver">
    <property name="prefix" value="/"/>
    <property name="suffix" value=".jsp"/>
</bean>

This worked and I was able to successfully see "Hello World" from a JSP in my browser. FreeMarker/SiteMesh worked too, but FreeMarker didn't work as a View for Spring MVC.

To attempt to solve this, I create a bundle for FreeMarker using "java -jar bnd-0.0.249.jar wrap freemarker-2.3.12.jar" and installed it in Equinox. I then change my MANIFEST.MF to use FreeMarker imports instead of referencing the JAR in WEB-INF/lib.

Bundle-Classpath:
.,WEB-INF/classes,WEB-INF/lib/sitemesh-2.3.jar,WEB-INF/lib/urlrewritefilter-3.0.4.jar
Import-Package:
javax.servlet,javax.servlet.http,javax.servlet.resources,javax.swing.tree,
 javax.naming,org.w3c.dom,org.apache.commons.logging,javax.xml.parsers;resolution:=optional,
 org.xml.sax;resolution:=optional,org.xml.sax.helpers;resolution:=optional,
 org.springframework.osgi.web.context.support,
 org.springframework.context.support,
 org.springframework.web.context,
 org.springframework.web.context.support,
 org.springframework.web.servlet,
 org.springframework.web.servlet.mvc,
 org.springframework.web.servlet.mvc.support,
 org.springframework.web.servlet.view,
 org.springframework.ui,
 org.springframework.web.servlet.view.freemarker,
 freemarker.cache,freemarker.core,freemarker.template,freemarker.ext.servlet 

Unfortunately, this still doesn't work and I still haven't been able to get FreeMarker to work with Spring MVC in OSGi. The crazy thing is I actually solved this at one point a week ago. Shortly after, I rebuilt Equinox from scratch and I'm been banging my head against the wall over this issue ever since. Last week, I entered an issue in Spring's JIRA, but thought I'd fixed it a few hours later.

I've uploaded the final project that's not working to the following URL:

http://static.raibledesigns.com/downloads/myapp-osgi.zip

If you'd like to see this project work with Spring MVC + JSP, simply modify myapp-servlet.xml to remove the FreeMarker references and use the InternalResourceViewResolver instead.

I hope Spring DM + Spring MVC supports more than just JSP as a view technology. I hope I can't get FreeMarker working because of some oversight on my part. If you have a Spring DM + Spring MVC application working with Velocity or FreeMarker, I'd love to hear about it.

Maven Integration for Eclipse

If you're a Maven user and like Eclipse, you might want to checkout the new Maven Integration Plugin for Eclipse 0.9.0. Euxx has a couple blog posts talking about the new features - looks like pretty cool stuff to me.

• Indexing Maven repositories - for offline browsing of repositories.
• Making a good use of Maven metadata - materializing projects from dependencies.
  

The second feature is especially cool. If your dependencies supply SCM information - you can import the project from its source control system. Maven may have warts, but it also has incredible potential.

Grails vs. Rails - My Thoughts

In a comment, Jared Peterson asked:

I'm curious if you have any thoughts on folks that might be trying to make a decision between Rails and Grails. I like the concept of "Allow Both", but what if you "have neither"?

If you were starting a new project, could choose either one, needed to interact with a lot of existing Java code (JRuby on Rails I guess), what would you pick?

A friend recently asked me "Can I solicit your honest, unadulterated opinion on Grails?" I think the e-mail I sent him may help Jared's question.

I think it's awesome. IMO, it's the same thing as AppFuse, but it has a DSL that's much simpler to learn and remember. Less code -> faster productivity. There does seem to be some maturity issues, but I think it'll get there. The question is - how fast can Groovy become. It's similar to Rails and Ruby in that you start using Grails and you think "This Groovy thing is kinda cool, I'd like to learn more." One of the reasons I really like it is the learning curve for experienced open source Java Developers is virtually flat. You can learn enough to be productive in a single day.

That being said, I think there's also a lot of cool stuff going on with RIA. IMO, Flex or GWT + Grails would be a really fun set of tools to develop with. Here's a excerpt from a write-up I recently did when analyzing Rails and Grails at LinkedIn (in January):

--------
Comparing Rails and Grails
They're both excellent frameworks. Rails is definitely more mature, but the environment is a pain to setup (esp. on Windows). Grails is very easy to setup for Java Developers. Grails needs a lot of improvement as far as hot deploy and stack traces. It's probably Groovy's fault, but its stack traces are hideous - rarely pointing to the class and line number in the first few lines.

As for hot deploy, it doesn't work nearly as well as it does with Rails. Rails' "script/server" starts WEBrick in a few seconds, while "grails run-app" can take up to 10 seconds (even on a brand new application). Even with its warts, Grails is simply awesome. I really, really enjoy writing Groovy code in IDEA and seeing immediate changes. I don't like "test-app" as much as I like Rails' "test:units" (or even better, "test:uncommitted"). It seems to be widely realized that Rails has a better testing story.

Rails is immediate, Grails is immediate 70% of the time.

Groovy is extremely easy to learn for Java Developers. Ruby is easy too learn, and possibly too powerful for OO rookies. Both are fun to program in and very capable of allowing greater developer productivity. If you know Hibernate, Spring, SiteMesh and JSP, you owe it to yourself to look at Grails. If you know these technologies well, you can learn Grails in less than an hour. You can be productive in the next hour and have an application running by the end of the day. That's not to take anything away from Ruby. I believe that Rails is an excellent platform as well. It's pretty cool that profiling and benchmarking are built into the framework and you can easily judge how many servers you'll need to scale.

I used IDEA while developing with both frameworks. IDEA has Rails and Groovy support available via plugins and they both worked quite well. The support for Grails was much better than Rails. Grails offers code completion, Ctrl+click on classes/methods, debugging and starting/ stopping the webapp from your IDE. Rails doesn't offer much in the way of Ctrl+clicking on class names/methods or debugging.
--------

Is there anything that Rails can do that Grails can't? Not as far as I can tell. I think it really comes down to developer passion and team preference. If you have experienced Java Developers that like the ecosystem and its tools, Grails makes a lot of sense. If you have experienced PHP developers or frustrated J2EE developers, they might enjoy Rails more. One thing that's very cool about both frameworks - learning one actually teaches you things about the other. They're so similar in many respects that knowledge is transferable between the two.

Of course, this is all just my opinion after working with both frameworks for a few weeks. For anyone who has tried both, what do you think?

In closing, here's an excerpt from a recent comment I left on Javalobby:

Of course, the hard part now is deciding between Django, Rails, Grails and GWT for your web framework. Then again, that's like having to choose between a Ferrari, Porsche, Lamborghini and a Maserati. No matter which one you choose, it's unlikely you'll be disappointed.

Last Night's Selenium Users Meetup at Google

Last night, I attended the inaugural Selenium User Group meetup at Google's Campus in Mountain View. It was an excellent event, with many of the core committers on hand to present and answer questions. Each presenter had about 5 minutes to speak and we learned many things about the Selenium Project itself, what's coming in the future and how Google has standardized on Selenium as their integration testing tool of choice.

Patrick Lightbody started the meeting by going over a number of project statistics with pretty graphs and such. There were simply too many numbers to write down, so hopefully his slides will be published soon. I was pleased to see that Google did videotape the entire event, so it should be available online soon. I'll update this post when it it. Below are my notes from the event.

Jason Huggins, Test Engineer at Google
Selenium is a test automation framework. Some folks abuse it as a macro tool. There's two reasons Selenium became so popular: it was able to test Ajax before any other testing tool and it allows end-to-end workflow testing. Selenium works on any platform, with any browser and allows many, many languages. It's possible that other frameworks that are more focused are better. There are 4 Selenium products:

• Selenium Core (TestRunner)
• Selenium IDE (for Firefox)
• Selenium Remote Control
• Selenium Grid

Paul Hammant, Open Source Manager at Thoughtworks
Selenium 1.0 - beta this week. RC, Core, Grid and IDE together. 1.0 will be shipped in a few weeks. Compared to today, lots of bugs killed, documentation improved and a greatly improved Selenium IDE.

Selenium 1.1 will be shipped in a couple months. Selenium IDE will be enhanced to obey RC instruction ... becoming the best mode of operation when it ships.

Some experimental iPhone-Safari capability (dependent on SDK for iPhone).

Selenium 2.0 will hopefully be released this year. Roll in of WebDriver functionality and code. New model-based API. IE plugin and Safari plugin (not really a plugin, most likely uses AppleScript).

Philippe Hanrigou (http://ph7spot.com)
Testing is good! Selenium is awesome! However, end-to-end web testing is slow. It is and always will be slow. How do you solve this problem? You solve it the same way we've solved slow traffic in the past - by building more lanes. Rather than one browser testing everything, Selenium Grid allows multiple lanes and 20 browsers. Features include:

• Faster Builds, Faster Feedback!
• Easy Install and Everyday Use
• No Need to Change Your Tests
• Leverage Your Existing Computing Infrastructure

One of the best parts about Selenium Grid is you can download and have it running in 10 minutes or less. Selenium Grid comes out-of-the-box with Amazon EC2 support.

Jennifer Bevan, Lead from Selenium Farm Project at Google
As of Friday, Google has over 50 teams running over 51K tests per day on internal Selenium Farm. 96% of these tests are handled by Selenium RC and the Farm machines correctly. The other 4% are partly due to RC bugs, partly to test errors, but isolating the cause can be difficult. Selenium has been adopted as the primary technology for functional testing of web applications within Google. That's the good news.

The bad news is Google is pushing the limits of Selenium. Using Selenium (RC + Core) at this scale exposed certain issues not originally anticipated as high-impact. IE/XPath slowness has a high impaction given that can only run one test at a time. Tests can cause many conditions from which RC cannot easily or automatically recover (for example, tests that don't call stop() in every exit path). Unexpected browser dialogs, popups, etc. eventually cause timeout exceptions.

Googlers expect that RC will work for the most part, but they want it to be more reliable, with better performance. So they have:

1. Created utility methods to improve performance when examinging large tables, overlaying domain-specific languages, etc.
2. Deployed retry policies based on failure reason.

Looking to the future - Google has not yet reached their expected usage of Selenium RC. Some projects cannot use the Farm until RC supports session-level configuration (not server-level). Many just want RC to be more reliable. So Google will:

1. Continue to contribute to RC, Core and to user-created helper libraries.
2. Keep doing so until all failed tests are not Selenium's fault.

Dave Astels, Google (Driving Selenium with RSpec)
Using RSpec, you can create a very easy-to-read Story with Scenarios that can be read (and likely written) by practically anyone. Dave then uses a small script to load up the stories and run them in Selenium. When he runs the script, the scenario is spit out and test pass/fail information. Learn more at rspec.info.

Alex Chaffee, Mad Scientist at Pivotal Labs (The Selenium/Ruby Project that Must Not Be Named)
What is Polonium? It's also known as Selenium RC Fu or Selenium On Rails 2 or Funkytown. It has simple extensions to Selenium RC:

• wait_for
• element assertions
• launching/managing servers locally

Blackbox testing you're sitting out of the box and send in stuff. In whitebox testing, you get to open up the box and look at stuff. With Selenium, you can do Graybox testing, where you are doing blackbox testing (against the UI) and querying your database (or other resources) at the same time.

Dan Faulich, Sr. QA Engineer at Redfin Corporation (How Not to Run a Successful Open Source Project)
The first thing you don't want to do is support everything. The second thing you don't want to do is write your project in JavaScript. While it's great that almost anyone can hack on it, running in a sandbox sucks. Don't use a language that's bound by other people's security restrictions. Don't roll your own multi-language remoting. It's written using XML + XSLT and its such a pain in the ass to maintain that Dan is the only one that fixes bugs.

Shinya Kasatani, Developer of Selenium IDE
Selenium IDE is a Firefox extension that can record and play back tests in your browser. It can translate the recorded tests to many languages.

Selenium IDE 1.0 adds support for TestSuites. Another new feature is better recording features - it detects when the DOM is modified. Shinya has a demo where he uses the new IDE to test the Dojo Dijit Theme Test Page. Apparently, this doesn't work in the current version.

The Goal of Selenium IDE is to get more people interested in test automation of web applications and to help their projects to be successful.

Haw-bin Chai, Developer at CommerceHub
XPath is a powerful selection took, but it'd be great if we could use something like "article 5" instead of the cryptic //table[5]/tbody/... syntax.

Why UI-Element? Traditional locations can be ugly and they don't convey purpose. UI-Element is a Selenium Core extension and has Selenium IDE integration. It's written in 100% JavaScript. Examples:

ui=frontPage::topStoriesCountry()
ui=listingPages::article(index=5)
ui=listingPages::articleSource(articleIndex=1)

The UI Map is in JavaScript shorthand. It contains logic to map ui locators to traditional locations. Each UI element implements getLocator(), then you add a page set and add an element - all in JSON Format.

Simon Stewart on Web Driver
What on earth is Web Driver? One of the main problems with Selenium is it's written in JavaScript and it's too easy to hack. WebDriver is an idealized browser which allows you to do browser things like call get(URL), findElement(By.xpath()) and getTitle(). WebDriver is similar to Selenium RC, but it's not written using JavaScript. It's written in the native languages for each browser, which allows you to break out of the JavaScript Jail. The IE Driver is written in COM (C++). The Firefox Driver is written as a Firefox extension. Safari uses Apple events.

Soon there won't be a WebDriver project ... because it will be part of Selenium!

WebDriver likely won't be part of the core until Selenium 2.0 later this year. One of the nice things about WebDriver is you can implement different browsers. Out-of-the-box, it will support all the major browsers, as well as HtmlUnit with JavaScript turned off.

I had a great time learning more about Selenium and how most of its problems will be solved in the near future. The beers afterwards weren't so bad either.

Update: Videos of this event have been posted.