Raible Designs

Building LinkedIn's Next Generation Architecture with OSGi by Yan Pujante

This week, I'm attending the Colorado Software Summit in Keystone, Colorado. Below are my notes from an OSGi at LinkedIn presentation I attended by Yan Pujante.

LinkedIn was created in March of 2003. Today there's close to 30M members. In the first 6 months, there was 60,000 members that signed up. Now, 1 million sign up every 2-3 weeks. LinkedIn is profitable with 5 revenue lines and there's 400 employees in Mountain View.

Technologies: 2 datacenters (~600 machines). SOA with Java, Tomcat, Spring Framework, Oracle, MySQL, Servlets, JSP, Cloud/Graph, OSGi. Development is done on Mac OS X, production is on Solaris.

The biggest challenges for LinkedIn are:

• Growing Engineering Team on a monolithic code base (it's modular, but only one source tree)
• Growing Product Team wanting more and more features faster
• Growing Operations Team deploying more and more servers

Front-end has many BL services in one webapp (in Tomcat). The backend is many wars in 5 containers (in Jetty) with 1 service per WAR. Production and Development environments are very different. Total services in backend is close to 100, front-end has 30-40.

Container Challenges
1 WAR with N services does not scale for developers (conflicts, monolithic). N wars with 1 service does not scale for containers (no shared JARs). You can add containers, but there's only 12GB of RAM available.

Upgrading back-end service to new version requires downtime (hardware load-balancer does not account for version). Upgrading front-end service to new version requires redeploy. Adding new backend services is also painful because there's lots of configuration (load-balancer, IPs, etc.).

Is there a solution to all these issues? Yan believes that OSGi is a good solution. OSGi stands for Open Services Gateway initiative. Today that term doesn't really mean anything. Today it's a spec with several implementations: Equinox (Eclipse), Knoplerfish and Felix (Apache).

OSGi has some really, really good features. These include smart class loading (multiple versions of JARs is OK), it's highly dynamic (deploy/undeploy built-in), it has a service registry and is highly extensible/configurable. An OSGi bundle is simply a JAR file with an OSGi manifest.

In LinkedIn's current architecture, services are exported with Spring/RPC and services in same WAR can see each other. The problem with this architecture comes to light when you want to move services to a 2nd web container. You cannot share JARs and can't talk directly to the other web app. With OSGi, the bundles (JARs) are shared, services are shared and bundles can be dynamically replaced. OSGi solves the container challenge.

One thing missing from OSGi is allowing services to live on multiple containers. To solve this, LinkedIn has developed a way to have Distributed OSGi. Multicast is used to see what's going on in other containers. Remote servers use the OSGi lifecycle and create dynamic proxies to export services using Spring RPC over HTTP. Then this service is registered in the service registry on the local server.

With Distributed OSGi, there's no more N-1 / 1-N problem. Libraries and services can be shared in one container (memory footprint is much smaller). Services can be shared across containers. The location of the services is transparent to the clients. There's no more configuration to change when adding/removing/moving services. This architecture allows the software to be the load balancer instead of using a hardware load balancer.

Unfortunately, everything is not perfect. OSGi has quite a few problems. OSGi is great, but the tooling is not quite there yet. Not every library is a bundle and many JARs doesn't have OSGi manifests. OSGi was designed for embedded devices and using it for the server-side is very recent (but very active).

OSGi is pretty low-level, but there is some work being done to hide the complexity. Spring DM helps, as do vendor containers. SpringSource has Spring dm Server, Sun has GlassFish, and Paremus has Infiniflow. OSGi is container centric, but next version will add distributed OSGi, but will have no support for load-balancing.

Another big OSGi issue is version management. If you specify version=1.0.0, it means [1.0.0, ∞]. You should at least use version=[1.0.0,2.0.0]. When using OSGi, you have to be careful and follow something similar to Apache's APR Project Versioning Guidelines so that you can easily identify release compatibility.

At LinkedIn, the OSGi implementation is progressing, but there's still a lot of work to do. First of all, a bundle repository needs to be created. Ivy and bnd is used to generate bundles. Containers are being evaluated and Infiniflow is most likely the one that will be used. LinkedIn Spring (an enhanced version of Spring) and SCA will be used to deploy composites. Work on load-balancing and distribution is in progress as is work on tooling and build integration (Sigil from Paremus).

In conclusion, LinkedIn will definitely use OSGi but they'll do their best to hide the complexity from the build system and from developers. For more information on OSGi at LinkedIn, stay tuned to the LinkedIn Engineering Blog. Yan has promised to blog about some of the challenges LinkedIn has faced and how he's fixed them.

Update: Yan has posted his presentation on the LinkedIn Engineering Blog.

The Colorado Software Summit and Spring 3.0's SVN

In a little over two weeks, my favorite conference begins in Keystone, Colorado. The reason I like it so much is because it's an annual gathering (this will be my 4th year) with friends and it's somewhat like a vacation, except you get to learn a lot. This year I'll be speaking about Appcelerator and Spring 3.0.

A few weeks ago, I wondered How open source is Spring and expressed my frustration in not being able to find the source code. At almost the same time, SpringSource's controversial maintenance policy was announced. Developers booed, Free Spring was founded and SpringSource changed their tune. In addition, someone sent me the FishEye URL for Spring 3.0's SVN. From that, I was able to figure out how to get Spring 3.0's source code.

svn co https://src.springframework.org/svn/spring-framework/trunk spring-3.0

The only bad news is FishEye shows the last change as "17 September 2008 ... (21 days ago)". If Spring 3.0 M1 is supposed to be released any day now, you'd think there would be more SVN commits. Anyone from SpringSource care to comment?

Xebia RIA Framework Contest

Last year, I blogged about the Xebia Web Framework Contest where Struts 2, GWT, Wicket and MyFaces (JSF) were all used to develop the same applications. It seems they've done it again this year, comparing RIA frameworks (English translation) this time.

The winner? Flex - which doesn't surprise me a whole lot.

After this day, the Flex team, has clearly demarcated from its competitors. After two sprints of development and ownership, it could devote the third and final sprint to get rich quick and easy application, focusing only on the features and user experience. It is thanks to the wealth of high-level components, the wealth of documentation available and a maturity framework that the team has made the Flex application's most successful.

Good stuff - thanks Xebia!

Ajaxified Body: Now with Back Button and Bookmarking Support

I uploaded a new version of Ajaxified Body to its demo site a few minutes ago. Changelog:

Added RSH for history and bookmarking and changed to use Prototype and Scriptaculous from Google's CDN.

I found Really Simple History easy to use but poorly documented. Brad Neuberg's AJAX: How to Handle Bookmarks and Back Buttons article was the one that got me over the learning curve. Hat tip to Ignacio for the RSH suggestion.

Let me know if you see any issues or have suggestions for improvement.

Ajaxified Body

I've often wondered if it was possible to use Ajax to reload the main content of a web application without reloading the header, menu and footer. SiteMesh allows you to move these common elements to a decorator that gets wrapped around each page. Below is a diagram of how SiteMesh works.

You can read the Introduction to SiteMesh article if you're interested in learning more about SiteMesh's basic features. By default, SiteMesh decorates text/html responses and ignores any other content type (e.g. image/gif). It also contains an <excludes> configuration element that allows you to turn off decoration when a URL matches a certain pattern. For example, the following allows you to disable decoration when "ajax=true" is passed in the URL.

<excludes>
    <pattern>**ajax=true</pattern>
</excludes>

To optimize the loading of an application so the common elements aren't loaded for each page, it should be possible to create an Ajaxified Body where the primary content area (PCA) of the site is loaded via Ajax rather than refreshing the whole page. The header, footer and navigation bar often stays the same from page-to-page, so it doesn't seem to make a whole lot of sense to load them each time the page loads. The image below shows the PCA (of an AppFuse Light application) as a grey square with a red border.

Implementing an Ajaxified Body consists of the following steps:

1. Adding SiteMesh and moving common elements to a decorator.
2. Remove common elements from each individual page (if you're using includes).
3. Configure SiteMesh so decoration is disabled when the requested URL contains "ajax=true".
4. Write JavaScript that modifies all <a href=""> links (and buttons with onclick='location.href') in the PCA to have an onclick handler.
5. The onclick handler should call a JavaScript function that loads the link's URL + ajax=true using XMLHttpRequest (XHR).
6. Add XHR success handling to replace the PCA with the loaded content.
7. Add XHR error handling to go to the URL normally when response.status != 200.
8. Inspect the response HTML for <title> element and replace document.title if exists.
9. Inspect the response HTML for <head> element and append to current if exists.
10. Inspect the response HTML for <script> and <link> elements (JavaScript and CSS) and evaluate them if they exist.

As a proof of concept, I created a prototype using AppFuse Light (Prototype/Scriptaculous for Ajax). You can see a demo at the following URL. You can also download a patch or the source for this project.

http://demo.raibledesigns.com/ajaxifiedbody

Below are a number of things I discovered while writing this prototype:

• The hardest part of implementing this seems to be coding the exceptions. It's possible you'll have some links with existing onclick handlers and you may have to disable "ajaxifying links" for those links.
• A progress indicator is important or the page might load so fast that the user doesn't visually detect it changed. This can lead to a worse user experience because they don't see the flash of the blank page they're used to when a page load occurs.
• While forms can be submitted via Ajax, there's no harm in leaving existing form behavior in place where the full site is reloaded after submitting a form.
• If a particular page needs to change the common elements (header, menu, footer), it should be possible to do that with JavaScript after the PCA content loads.
• If the success/error indicator is outside the PCA, it may need to be populated and displayed/hidden with JavaScript after the PCA loads.

I'm sure my implementation can be improved, but I'm also curious to see what you think of this idea. I know it's not revolutionary, but it's something I'm considering adding by default to AppFuse and AppFuse Light. Do any Ajax frameworks do something like this out-of-the-box?

Update: Thanks to everyone for the great feedback - keep it coming. I agree that adding history support is a must. I'll try to do that in the next day or two. This post has also been featured on Javalobby and Ajaxian.

Update 2: Added history support.