Raible Designs

Building LinkedIn's Next Generation Architecture with OSGi by Yan Pujante

This week, I'm attending the Colorado Software Summit in Keystone, Colorado. Below are my notes from an OSGi at LinkedIn presentation I attended by Yan Pujante.

LinkedIn was created in March of 2003. Today there's close to 30M members. In the first 6 months, there was 60,000 members that signed up. Now, 1 million sign up every 2-3 weeks. LinkedIn is profitable with 5 revenue lines and there's 400 employees in Mountain View.

Technologies: 2 datacenters (~600 machines). SOA with Java, Tomcat, Spring Framework, Oracle, MySQL, Servlets, JSP, Cloud/Graph, OSGi. Development is done on Mac OS X, production is on Solaris.

The biggest challenges for LinkedIn are:

• Growing Engineering Team on a monolithic code base (it's modular, but only one source tree)
• Growing Product Team wanting more and more features faster
• Growing Operations Team deploying more and more servers

Front-end has many BL services in one webapp (in Tomcat). The backend is many wars in 5 containers (in Jetty) with 1 service per WAR. Production and Development environments are very different. Total services in backend is close to 100, front-end has 30-40.

Container Challenges
1 WAR with N services does not scale for developers (conflicts, monolithic). N wars with 1 service does not scale for containers (no shared JARs). You can add containers, but there's only 12GB of RAM available.

Upgrading back-end service to new version requires downtime (hardware load-balancer does not account for version). Upgrading front-end service to new version requires redeploy. Adding new backend services is also painful because there's lots of configuration (load-balancer, IPs, etc.).

Is there a solution to all these issues? Yan believes that OSGi is a good solution. OSGi stands for Open Services Gateway initiative. Today that term doesn't really mean anything. Today it's a spec with several implementations: Equinox (Eclipse), Knoplerfish and Felix (Apache).

OSGi has some really, really good features. These include smart class loading (multiple versions of JARs is OK), it's highly dynamic (deploy/undeploy built-in), it has a service registry and is highly extensible/configurable. An OSGi bundle is simply a JAR file with an OSGi manifest.

In LinkedIn's current architecture, services are exported with Spring/RPC and services in same WAR can see each other. The problem with this architecture comes to light when you want to move services to a 2nd web container. You cannot share JARs and can't talk directly to the other web app. With OSGi, the bundles (JARs) are shared, services are shared and bundles can be dynamically replaced. OSGi solves the container challenge.

One thing missing from OSGi is allowing services to live on multiple containers. To solve this, LinkedIn has developed a way to have Distributed OSGi. Multicast is used to see what's going on in other containers. Remote servers use the OSGi lifecycle and create dynamic proxies to export services using Spring RPC over HTTP. Then this service is registered in the service registry on the local server.

With Distributed OSGi, there's no more N-1 / 1-N problem. Libraries and services can be shared in one container (memory footprint is much smaller). Services can be shared across containers. The location of the services is transparent to the clients. There's no more configuration to change when adding/removing/moving services. This architecture allows the software to be the load balancer instead of using a hardware load balancer.

Unfortunately, everything is not perfect. OSGi has quite a few problems. OSGi is great, but the tooling is not quite there yet. Not every library is a bundle and many JARs doesn't have OSGi manifests. OSGi was designed for embedded devices and using it for the server-side is very recent (but very active).

OSGi is pretty low-level, but there is some work being done to hide the complexity. Spring DM helps, as do vendor containers. SpringSource has Spring dm Server, Sun has GlassFish, and Paremus has Infiniflow. OSGi is container centric, but next version will add distributed OSGi, but will have no support for load-balancing.

Another big OSGi issue is version management. If you specify version=1.0.0, it means [1.0.0, ∞]. You should at least use version=[1.0.0,2.0.0]. When using OSGi, you have to be careful and follow something similar to Apache's APR Project Versioning Guidelines so that you can easily identify release compatibility.

At LinkedIn, the OSGi implementation is progressing, but there's still a lot of work to do. First of all, a bundle repository needs to be created. Ivy and bnd is used to generate bundles. Containers are being evaluated and Infiniflow is most likely the one that will be used. LinkedIn Spring (an enhanced version of Spring) and SCA will be used to deploy composites. Work on load-balancing and distribution is in progress as is work on tooling and build integration (Sigil from Paremus).

In conclusion, LinkedIn will definitely use OSGi but they'll do their best to hide the complexity from the build system and from developers. For more information on OSGi at LinkedIn, stay tuned to the LinkedIn Engineering Blog. Yan has promised to blog about some of the challenges LinkedIn has faced and how he's fixed them.

Update: Yan has posted his presentation on the LinkedIn Engineering Blog.

The Colorado Software Summit and Spring 3.0's SVN

In a little over two weeks, my favorite conference begins in Keystone, Colorado. The reason I like it so much is because it's an annual gathering (this will be my 4th year) with friends and it's somewhat like a vacation, except you get to learn a lot. This year I'll be speaking about Appcelerator and Spring 3.0.

A few weeks ago, I wondered How open source is Spring and expressed my frustration in not being able to find the source code. At almost the same time, SpringSource's controversial maintenance policy was announced. Developers booed, Free Spring was founded and SpringSource changed their tune. In addition, someone sent me the FishEye URL for Spring 3.0's SVN. From that, I was able to figure out how to get Spring 3.0's source code.

svn co https://src.springframework.org/svn/spring-framework/trunk spring-3.0

The only bad news is FishEye shows the last change as "17 September 2008 ... (21 days ago)". If Spring 3.0 M1 is supposed to be released any day now, you'd think there would be more SVN commits. Anyone from SpringSource care to comment?

Appcelerator with Matt Quinlan

This evening I attended the Denver Open Source User Group meeting where the Basic Concepts talk was on Appcelerator. Matt Quinlan (Twitter, Blog, LinkedIn) was the presenter. I arrived 10 minutes late, so I didn't hear any of the intro stuff. Below are my notes from the event.

The Appcelerator developers liked the "onclick" syntax, but found it was too limited to do everything they wanted. Rather than onclick, they use an on attribute. For example:

<div on="click then l:show.box">
Click me
<div>

"DOM Manipulation is JavaScript cruft."

Appcelerator allows you to implement the Observer pattern in the browser. In addition to allowing DOM elements to subscribe to messages, server-side objects (in any language) can subscribe as well. In Appcelerator syntax "l:" means local and "r:" means remote. The messages that are passed to the server are JSON and have payloads. JSON is more popular than XML because you can eval() it and create JavaScript objects from it. Appcelerator allows you to do Declarative Ajax. On the server-side, you can annotation Java and C# classes with @Service to subscribe to messages. In other languages (i.e. PHP), Javadoc-style comments are used.

Tagline: The seamless fusion of RIA and SOA.

Web Expression Language
Goal is to eliminate 90% of the JavaScript you write. Example syntax:

on="[condition] then [action]"

Conditions include DOM events (click, focus, blur, change, mouse events), key events (up, down, press), other (history, drag/drop, selected, resize, iPhone orient, sortXYZ), subscribe to custom message. Actions include Scriptaculous effects (show/hide, fade, move, drop, grow), set element value (static, dynamic, bind), set CSS class or attribute, execute custom JavaScript, publish custom message.

Now Matt is doing a demo on http://try.appcelerator.org. This site consists of a form that allows you to type in Appcelerator code and run it. 3 attributes can be added to any tag: draggable, droppable and resizeable.

Client-Side
Add simple tags to your HTML to inject RIA widgets. Add single property to existing HTML elements for dynamic behavior. Eclipse Plugin built on Aptana, but is generally targeted towards web developers moreso than business analysts (no drag and drop of widgets).

Server-Side
Server-side development done with your IDE of choices. Based on your server-side tehchnology platform. Easily create services using annotations.

App Command
The app command is similar to Rails' GEM command. Allows you to build scaffolding and deploy to cloud (AppEngine, Amazon S3). It also allows you to pulldown components from the main server and auto-updating.

Examples:

• appTunes: widget that wraps Flex Cover Flow widget
• Snapshot: photo editing application
• Radio Javan: Persian music online
• SkyBlox: wireless company

Appcelerator allows you to create prototypes easily by using a JavaScript file with mocks for the server-side objects. In the next version, you can "annotate" the UI and allow end-users to Ctrl+Click on elements and add feedback. For an example of this, see http://dev.appcelerator.com/pearson.

When starting with Appcelerator, you can start by crawling (including appcelerator.js for widgets) then move to walking (decouple server-side and client-side) and finally running (developing working prototypes with mocks for server-side).

"Let's face it, ASP, JSP, PHP and Ruby are just lipstick on CGI."

Xebia RIA Framework Contest

Last year, I blogged about the Xebia Web Framework Contest where Struts 2, GWT, Wicket and MyFaces (JSF) were all used to develop the same applications. It seems they've done it again this year, comparing RIA frameworks (English translation) this time.

The winner? Flex - which doesn't surprise me a whole lot.

After this day, the Flex team, has clearly demarcated from its competitors. After two sprints of development and ownership, it could devote the third and final sprint to get rich quick and easy application, focusing only on the features and user experience. It is thanks to the wealth of high-level components, the wealth of documentation available and a maturity framework that the team has made the Flex application's most successful.

Good stuff - thanks Xebia!

Ajaxified Body

I've often wondered if it was possible to use Ajax to reload the main content of a web application without reloading the header, menu and footer. SiteMesh allows you to move these common elements to a decorator that gets wrapped around each page. Below is a diagram of how SiteMesh works.

You can read the Introduction to SiteMesh article if you're interested in learning more about SiteMesh's basic features. By default, SiteMesh decorates text/html responses and ignores any other content type (e.g. image/gif). It also contains an <excludes> configuration element that allows you to turn off decoration when a URL matches a certain pattern. For example, the following allows you to disable decoration when "ajax=true" is passed in the URL.

<excludes>
    <pattern>**ajax=true</pattern>
</excludes>

To optimize the loading of an application so the common elements aren't loaded for each page, it should be possible to create an Ajaxified Body where the primary content area (PCA) of the site is loaded via Ajax rather than refreshing the whole page. The header, footer and navigation bar often stays the same from page-to-page, so it doesn't seem to make a whole lot of sense to load them each time the page loads. The image below shows the PCA (of an AppFuse Light application) as a grey square with a red border.

Implementing an Ajaxified Body consists of the following steps:

1. Adding SiteMesh and moving common elements to a decorator.
2. Remove common elements from each individual page (if you're using includes).
3. Configure SiteMesh so decoration is disabled when the requested URL contains "ajax=true".
4. Write JavaScript that modifies all <a href=""> links (and buttons with onclick='location.href') in the PCA to have an onclick handler.
5. The onclick handler should call a JavaScript function that loads the link's URL + ajax=true using XMLHttpRequest (XHR).
6. Add XHR success handling to replace the PCA with the loaded content.
7. Add XHR error handling to go to the URL normally when response.status != 200.
8. Inspect the response HTML for <title> element and replace document.title if exists.
9. Inspect the response HTML for <head> element and append to current if exists.
10. Inspect the response HTML for <script> and <link> elements (JavaScript and CSS) and evaluate them if they exist.

As a proof of concept, I created a prototype using AppFuse Light (Prototype/Scriptaculous for Ajax). You can see a demo at the following URL. You can also download a patch or the source for this project.

http://demo.raibledesigns.com/ajaxifiedbody

Below are a number of things I discovered while writing this prototype:

• The hardest part of implementing this seems to be coding the exceptions. It's possible you'll have some links with existing onclick handlers and you may have to disable "ajaxifying links" for those links.
• A progress indicator is important or the page might load so fast that the user doesn't visually detect it changed. This can lead to a worse user experience because they don't see the flash of the blank page they're used to when a page load occurs.
• While forms can be submitted via Ajax, there's no harm in leaving existing form behavior in place where the full site is reloaded after submitting a form.
• If a particular page needs to change the common elements (header, menu, footer), it should be possible to do that with JavaScript after the PCA content loads.
• If the success/error indicator is outside the PCA, it may need to be populated and displayed/hidden with JavaScript after the PCA loads.

I'm sure my implementation can be improved, but I'm also curious to see what you think of this idea. I know it's not revolutionary, but it's something I'm considering adding by default to AppFuse and AppFuse Light. Do any Ajax frameworks do something like this out-of-the-box?

Update: Thanks to everyone for the great feedback - keep it coming. I agree that adding history support is a must. I'll try to do that in the next day or two. This post has also been featured on Javalobby and Ajaxian.

Update 2: Added history support.

My Oktoberfest Adventure Begins

A few months ago, a friend called me up and asked if I wanted to go to Oktoberfest. He said he had enough miles to cover both our tickets. I couldn't pass up the opportunity and hence we're leaving at the crack of dawn tomorrow for a week in Munich, Germany. We're not taking laptops and we don't plan on doing a whole lot more than having a great time.

If you'd like to see what we're doing on our adventure, you can follow me on Twitter.

RE: How Open Source is Spring?

Peter Mularien has a very well written post titled How Open Source is Spring?: An Analytical Investigation:

This post is to expand on some of the thoughts I posted on the SpringSource Blog in response to Rod Johnson's excellent description of the SpringSource business model and its commitment to development of open source software.

Now that SpringSource has shown an ability to crank out new product releases on a seemingly weekly basis, I wanted to reflect on where Spring is positioned in the Java open source community, and how open the Spring Core project is to work done by the public.

The hypothesis of my experiment occurred to me when I happened to be reviewing Spring JIRA assignments one day. I was curious whether, following the bug assignments, the majority of development on the "Spring Core" projects (including Spring MVC and what we would consider "classic Spring") is performed solely by SpringSource employees.

Peter's post is near and dear to me because I'm doing a What's coming in Spring 3.0 talk at the Colorado Software Summit in October. The only information I was able to find on Spring 3.0 is from random blog posts and Juergen's presentation. When I give talks about technologies, I prefer to dig in and try them before talking about it. With Spring 3.0's source code nowhere to be found, this is very difficult to do.

I really hope Spring 3.0 becomes available in early October. If it does, I hope to upgrade AppFuse, AppFuse Light and Spring Kickstart. If it doesn't, my talk will most likely be a regurgitation of what Juergen's slides have and that's just not right.

My TripIt Blog Badge

Jared started it. TripIt provides it. Here's mine:

Upgrading Hibernate to 3.4.0 and AppFuse for Tapestry 5

Last night I spent an hour upgrading AppFuse to Hibernate 3.4.0.GA. I tried the same thing a week ago, but failed miserably. When Hibernate Core Modules (3.3) (a Maven dependency howto) showed up on the Hibernate blog, I was re-inspired. I discovered some interesting things along the way (at least in my setup).

You need to use JBoss's Maven Repository
The latest Hibernate releases aren't in the central Maven repo yet. If they're not in there two weeks after the release, I doubt they'll be there anytime soon. Best to plan on adding http://repository.jboss.com/maven2 as a permanent repository. As a bonus, you can remove http://download.java.net/maven/2 since JTA is in JBoss's repo.

Javassist is not an optional dependency
If I remove the Javassist dependency, here's the error I get:

java.lang.NoClassDefFoundError: javassist/util/proxy/MethodFilter

Seems reasonable right? What if I add in the dependency on Hibernate's cglib instead?

java.lang.NoClassDefFoundError: javassist/util/proxy/MethodFilter

OK, so apparently you can't switch between Javassist and hibernate-cglib-repack as stated in Hibernate's Maven dependency howto. Of course, I do believe their instructions are correct, they just don't work in my setup. Versions I'm using: hibernate-core-3.3.1.GA and hibernate-annotations-3.4.0.GA.

Not including SLF4J is a bad idea
If you don't include a dependency on SLF4J (and you're using Spring), you get a nice cryptic error message.

Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class 
[org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean]: 
Constructor threw exception; nested exception is java.lang.NoClassDefFoundError
        at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:115)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:61)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:877)
        ... 48 more
Caused by: java.lang.NoClassDefFoundError
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:164)
        at org.springframework.orm.hibernate3.LocalSessionFactoryBean.class$(LocalSessionFactoryBean.java:174)
        at org.springframework.orm.hibernate3.LocalSessionFactoryBean.(LocalSessionFactoryBean.java:174)
        at org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean.(AnnotationSessionFactoryBean.java:64)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:494)
        at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:100)

Hibernate's JPA now uses PersistenceException instead of EntityExistsException
In previous releases, Hibernate's ConstraintViolationException was wrapped in a javax.persistence.EntityExistsException. In Hibernate's EntityManager 3.4.0.GA, it's wrapped in a javax.persistence.PersistenceException.

Of course, these errors could be caused by Spring or Maven, but my hunch is they're more related to Hibernate and it's new more modular dependencies. You can view the full changeset for upgrading Hibernate 3.2.6.ga to 3.3.1.GA via FishEye.

Tapestry 5 version of AppFuse
In other AppFuse-related news, Serge Eby has created a Tapestry 5 version of AppFuse. You can see his alpha-level work in Google Code's tapestry5-appfuse project. I haven't had a chance to take a look at Serge's work yet, but I'm eager to do so. Hopefully we can get it back into the main project sooner than later. As far as Wicket and Stripes support, I haven't forgot about those - just having trouble finding the time and motivation to do the work.

Happy Birthday Jack!

Jack's birthday was last Thursday (August 28th) and only now am I getting a chance to wish him a Happy Birthday on this blog. I told you it was a busy week last week.

Happy Birthday Jack!

On his birthday, I picked him up from Kindergarten at 11 and let him run the show for the day. Needless to say, we had a good time.

On Saturday, we had a party to celebrate at my house. It was a great time with many old friends and lots of crazy kids.

I woke up to quite the disaster area the next day. I was also quite impressed when Abbie showed up with her ears pierced.

The rest of the weekend I spent quite a bit of time enjoying A Taste of Colorado. When work began on Tuesday, the week intensified as I was the "Release Owner" for LinkedIn this week. We made a number of backend changes to reduce statefulness and improve performance. I'm proud to say it was one of the smoothest releases I've been a part of, mostly because of the high quality people involved.

As far as my schedule for the rest of the year, it seems like I'm going to be busier than ever. I have to complete my presentations for CSS this week. In two weeks, I'm speaking at the Colorado Springs Open Source User Group Meetup. The following week I'm heading to Munich for Oktoberfest. If I live through that, I'll be implementing Memcached, speaking at CSS, hunting with my Dad, celebrating Abbie's birthday, traveling to Costa Rica/Panama and keynoting the JavaEdge Conference in Israel. If you happen to be in Colorado Springs, Munich, Mountain View, Keystone, Montana, Costa Rica, Panama or Israel in the next few months, maybe we can enjoy a beer together. Cheers!