Raible Designs

Building LinkedIn's Next Generation Architecture with OSGi by Yan Pujante

This week, I'm attending the Colorado Software Summit in Keystone, Colorado. Below are my notes from an OSGi at LinkedIn presentation I attended by Yan Pujante.

LinkedIn was created in March of 2003. Today there's close to 30M members. In the first 6 months, there was 60,000 members that signed up. Now, 1 million sign up every 2-3 weeks. LinkedIn is profitable with 5 revenue lines and there's 400 employees in Mountain View.

Technologies: 2 datacenters (~600 machines). SOA with Java, Tomcat, Spring Framework, Oracle, MySQL, Servlets, JSP, Cloud/Graph, OSGi. Development is done on Mac OS X, production is on Solaris.

The biggest challenges for LinkedIn are:

• Growing Engineering Team on a monolithic code base (it's modular, but only one source tree)
• Growing Product Team wanting more and more features faster
• Growing Operations Team deploying more and more servers

Front-end has many BL services in one webapp (in Tomcat). The backend is many wars in 5 containers (in Jetty) with 1 service per WAR. Production and Development environments are very different. Total services in backend is close to 100, front-end has 30-40.

Container Challenges
1 WAR with N services does not scale for developers (conflicts, monolithic). N wars with 1 service does not scale for containers (no shared JARs). You can add containers, but there's only 12GB of RAM available.

Upgrading back-end service to new version requires downtime (hardware load-balancer does not account for version). Upgrading front-end service to new version requires redeploy. Adding new backend services is also painful because there's lots of configuration (load-balancer, IPs, etc.).

Is there a solution to all these issues? Yan believes that OSGi is a good solution. OSGi stands for Open Services Gateway initiative. Today that term doesn't really mean anything. Today it's a spec with several implementations: Equinox (Eclipse), Knoplerfish and Felix (Apache).

OSGi has some really, really good features. These include smart class loading (multiple versions of JARs is OK), it's highly dynamic (deploy/undeploy built-in), it has a service registry and is highly extensible/configurable. An OSGi bundle is simply a JAR file with an OSGi manifest.

In LinkedIn's current architecture, services are exported with Spring/RPC and services in same WAR can see each other. The problem with this architecture comes to light when you want to move services to a 2nd web container. You cannot share JARs and can't talk directly to the other web app. With OSGi, the bundles (JARs) are shared, services are shared and bundles can be dynamically replaced. OSGi solves the container challenge.

One thing missing from OSGi is allowing services to live on multiple containers. To solve this, LinkedIn has developed a way to have Distributed OSGi. Multicast is used to see what's going on in other containers. Remote servers use the OSGi lifecycle and create dynamic proxies to export services using Spring RPC over HTTP. Then this service is registered in the service registry on the local server.

With Distributed OSGi, there's no more N-1 / 1-N problem. Libraries and services can be shared in one container (memory footprint is much smaller). Services can be shared across containers. The location of the services is transparent to the clients. There's no more configuration to change when adding/removing/moving services. This architecture allows the software to be the load balancer instead of using a hardware load balancer.

Unfortunately, everything is not perfect. OSGi has quite a few problems. OSGi is great, but the tooling is not quite there yet. Not every library is a bundle and many JARs doesn't have OSGi manifests. OSGi was designed for embedded devices and using it for the server-side is very recent (but very active).

OSGi is pretty low-level, but there is some work being done to hide the complexity. Spring DM helps, as do vendor containers. SpringSource has Spring dm Server, Sun has GlassFish, and Paremus has Infiniflow. OSGi is container centric, but next version will add distributed OSGi, but will have no support for load-balancing.

Another big OSGi issue is version management. If you specify version=1.0.0, it means [1.0.0, ∞]. You should at least use version=[1.0.0,2.0.0]. When using OSGi, you have to be careful and follow something similar to Apache's APR Project Versioning Guidelines so that you can easily identify release compatibility.

At LinkedIn, the OSGi implementation is progressing, but there's still a lot of work to do. First of all, a bundle repository needs to be created. Ivy and bnd is used to generate bundles. Containers are being evaluated and Infiniflow is most likely the one that will be used. LinkedIn Spring (an enhanced version of Spring) and SCA will be used to deploy composites. Work on load-balancing and distribution is in progress as is work on tooling and build integration (Sigil from Paremus).

In conclusion, LinkedIn will definitely use OSGi but they'll do their best to hide the complexity from the build system and from developers. For more information on OSGi at LinkedIn, stay tuned to the LinkedIn Engineering Blog. Yan has promised to blog about some of the challenges LinkedIn has faced and how he's fixed them.

Update: Yan has posted his presentation on the LinkedIn Engineering Blog.

The Colorado Software Summit and Spring 3.0's SVN

In a little over two weeks, my favorite conference begins in Keystone, Colorado. The reason I like it so much is because it's an annual gathering (this will be my 4th year) with friends and it's somewhat like a vacation, except you get to learn a lot. This year I'll be speaking about Appcelerator and Spring 3.0.

A few weeks ago, I wondered How open source is Spring and expressed my frustration in not being able to find the source code. At almost the same time, SpringSource's controversial maintenance policy was announced. Developers booed, Free Spring was founded and SpringSource changed their tune. In addition, someone sent me the FishEye URL for Spring 3.0's SVN. From that, I was able to figure out how to get Spring 3.0's source code.

svn co https://src.springframework.org/svn/spring-framework/trunk spring-3.0

The only bad news is FishEye shows the last change as "17 September 2008 ... (21 days ago)". If Spring 3.0 M1 is supposed to be released any day now, you'd think there would be more SVN commits. Anyone from SpringSource care to comment?

Xebia RIA Framework Contest

Last year, I blogged about the Xebia Web Framework Contest where Struts 2, GWT, Wicket and MyFaces (JSF) were all used to develop the same applications. It seems they've done it again this year, comparing RIA frameworks (English translation) this time.

The winner? Flex - which doesn't surprise me a whole lot.

After this day, the Flex team, has clearly demarcated from its competitors. After two sprints of development and ownership, it could devote the third and final sprint to get rich quick and easy application, focusing only on the features and user experience. It is thanks to the wealth of high-level components, the wealth of documentation available and a maturity framework that the team has made the Flex application's most successful.

Good stuff - thanks Xebia!

Ajaxified Body

I've often wondered if it was possible to use Ajax to reload the main content of a web application without reloading the header, menu and footer. SiteMesh allows you to move these common elements to a decorator that gets wrapped around each page. Below is a diagram of how SiteMesh works.

You can read the Introduction to SiteMesh article if you're interested in learning more about SiteMesh's basic features. By default, SiteMesh decorates text/html responses and ignores any other content type (e.g. image/gif). It also contains an <excludes> configuration element that allows you to turn off decoration when a URL matches a certain pattern. For example, the following allows you to disable decoration when "ajax=true" is passed in the URL.

<excludes>
    <pattern>**ajax=true</pattern>
</excludes>

To optimize the loading of an application so the common elements aren't loaded for each page, it should be possible to create an Ajaxified Body where the primary content area (PCA) of the site is loaded via Ajax rather than refreshing the whole page. The header, footer and navigation bar often stays the same from page-to-page, so it doesn't seem to make a whole lot of sense to load them each time the page loads. The image below shows the PCA (of an AppFuse Light application) as a grey square with a red border.

Implementing an Ajaxified Body consists of the following steps:

1. Adding SiteMesh and moving common elements to a decorator.
2. Remove common elements from each individual page (if you're using includes).
3. Configure SiteMesh so decoration is disabled when the requested URL contains "ajax=true".
4. Write JavaScript that modifies all <a href=""> links (and buttons with onclick='location.href') in the PCA to have an onclick handler.
5. The onclick handler should call a JavaScript function that loads the link's URL + ajax=true using XMLHttpRequest (XHR).
6. Add XHR success handling to replace the PCA with the loaded content.
7. Add XHR error handling to go to the URL normally when response.status != 200.
8. Inspect the response HTML for <title> element and replace document.title if exists.
9. Inspect the response HTML for <head> element and append to current if exists.
10. Inspect the response HTML for <script> and <link> elements (JavaScript and CSS) and evaluate them if they exist.

As a proof of concept, I created a prototype using AppFuse Light (Prototype/Scriptaculous for Ajax). You can see a demo at the following URL. You can also download a patch or the source for this project.

http://demo.raibledesigns.com/ajaxifiedbody

Below are a number of things I discovered while writing this prototype:

• The hardest part of implementing this seems to be coding the exceptions. It's possible you'll have some links with existing onclick handlers and you may have to disable "ajaxifying links" for those links.
• A progress indicator is important or the page might load so fast that the user doesn't visually detect it changed. This can lead to a worse user experience because they don't see the flash of the blank page they're used to when a page load occurs.
• While forms can be submitted via Ajax, there's no harm in leaving existing form behavior in place where the full site is reloaded after submitting a form.
• If a particular page needs to change the common elements (header, menu, footer), it should be possible to do that with JavaScript after the PCA content loads.
• If the success/error indicator is outside the PCA, it may need to be populated and displayed/hidden with JavaScript after the PCA loads.

I'm sure my implementation can be improved, but I'm also curious to see what you think of this idea. I know it's not revolutionary, but it's something I'm considering adding by default to AppFuse and AppFuse Light. Do any Ajax frameworks do something like this out-of-the-box?

Update: Thanks to everyone for the great feedback - keep it coming. I agree that adding history support is a must. I'll try to do that in the next day or two. This post has also been featured on Javalobby and Ajaxian.

Update 2: Added history support.

RE: How Open Source is Spring?

Peter Mularien has a very well written post titled How Open Source is Spring?: An Analytical Investigation:

This post is to expand on some of the thoughts I posted on the SpringSource Blog in response to Rod Johnson's excellent description of the SpringSource business model and its commitment to development of open source software.

Now that SpringSource has shown an ability to crank out new product releases on a seemingly weekly basis, I wanted to reflect on where Spring is positioned in the Java open source community, and how open the Spring Core project is to work done by the public.

The hypothesis of my experiment occurred to me when I happened to be reviewing Spring JIRA assignments one day. I was curious whether, following the bug assignments, the majority of development on the "Spring Core" projects (including Spring MVC and what we would consider "classic Spring") is performed solely by SpringSource employees.

Peter's post is near and dear to me because I'm doing a What's coming in Spring 3.0 talk at the Colorado Software Summit in October. The only information I was able to find on Spring 3.0 is from random blog posts and Juergen's presentation. When I give talks about technologies, I prefer to dig in and try them before talking about it. With Spring 3.0's source code nowhere to be found, this is very difficult to do.

I really hope Spring 3.0 becomes available in early October. If it does, I hope to upgrade AppFuse, AppFuse Light and Spring Kickstart. If it doesn't, my talk will most likely be a regurgitation of what Juergen's slides have and that's just not right.

Upgrading Hibernate to 3.4.0 and AppFuse for Tapestry 5

Last night I spent an hour upgrading AppFuse to Hibernate 3.4.0.GA. I tried the same thing a week ago, but failed miserably. When Hibernate Core Modules (3.3) (a Maven dependency howto) showed up on the Hibernate blog, I was re-inspired. I discovered some interesting things along the way (at least in my setup).

You need to use JBoss's Maven Repository
The latest Hibernate releases aren't in the central Maven repo yet. If they're not in there two weeks after the release, I doubt they'll be there anytime soon. Best to plan on adding http://repository.jboss.com/maven2 as a permanent repository. As a bonus, you can remove http://download.java.net/maven/2 since JTA is in JBoss's repo.

Javassist is not an optional dependency
If I remove the Javassist dependency, here's the error I get:

java.lang.NoClassDefFoundError: javassist/util/proxy/MethodFilter

Seems reasonable right? What if I add in the dependency on Hibernate's cglib instead?

java.lang.NoClassDefFoundError: javassist/util/proxy/MethodFilter

OK, so apparently you can't switch between Javassist and hibernate-cglib-repack as stated in Hibernate's Maven dependency howto. Of course, I do believe their instructions are correct, they just don't work in my setup. Versions I'm using: hibernate-core-3.3.1.GA and hibernate-annotations-3.4.0.GA.

Not including SLF4J is a bad idea
If you don't include a dependency on SLF4J (and you're using Spring), you get a nice cryptic error message.

Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class 
[org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean]: 
Constructor threw exception; nested exception is java.lang.NoClassDefFoundError
        at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:115)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:61)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:877)
        ... 48 more
Caused by: java.lang.NoClassDefFoundError
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:164)
        at org.springframework.orm.hibernate3.LocalSessionFactoryBean.class$(LocalSessionFactoryBean.java:174)
        at org.springframework.orm.hibernate3.LocalSessionFactoryBean.(LocalSessionFactoryBean.java:174)
        at org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean.(AnnotationSessionFactoryBean.java:64)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:494)
        at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:100)

Hibernate's JPA now uses PersistenceException instead of EntityExistsException
In previous releases, Hibernate's ConstraintViolationException was wrapped in a javax.persistence.EntityExistsException. In Hibernate's EntityManager 3.4.0.GA, it's wrapped in a javax.persistence.PersistenceException.

Of course, these errors could be caused by Spring or Maven, but my hunch is they're more related to Hibernate and it's new more modular dependencies. You can view the full changeset for upgrading Hibernate 3.2.6.ga to 3.3.1.GA via FishEye.

Tapestry 5 version of AppFuse
In other AppFuse-related news, Serge Eby has created a Tapestry 5 version of AppFuse. You can see his alpha-level work in Google Code's tapestry5-appfuse project. I haven't had a chance to take a look at Serge's work yet, but I'm eager to do so. Hopefully we can get it back into the main project sooner than later. As far as Wicket and Stripes support, I haven't forgot about those - just having trouble finding the time and motivation to do the work.

[DJUG] Ship it! and Distributed Teams by Jared Richardson

Jared's first book is called Ship it! and it's been wildly successful (translated into several different languages). During the first hour, Jared talked about his book and practices defined in it.

Ship it! is an attempt to talk about the holistic process of software development: techniques, infrastructure and process. For Techniques, there's a Technical Lead, The List, Code Reviews, Code Change Notifier and Daily Meetings. Infrastructure involves Version Control, Script Builds, Continuous Builds, Track Issues, Track Features and Writing/Running Tests. Process is Propose System Objects -> Propose Interfaces -> Connect Interfaces -> Add Functions -> Refactor, Refine, Repeat.

Martin Fowler's definition of Architecture: anything that is difficult to change later.

The majority of Jared's talk was about implementing TDD, continuous integration and many other things driven by audience questions. I didn't take notes because it was more of a conversation than a presentation. Jared's second talk was on Distributed Teams and my notes from this talk are below.

Agile is not XP or Scrum, those are implementations of Agile. The Manifesto for Agile Software Development is the interface for Agile Development. It contains the following principles:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Custom collaboration over contract negotiation
• Responding to change over following a plan

To re-word these principles: Agility is...

• People interacting
• Software that runs
• Fast Feedback
• Frequent Change

Scrum principles are often not about developing software, they're about making people talk. The important thing is to get people in the habit of communicating. The best way to get people interacting is to do Daily meetings. One to two minutes per person. Same time. Three questions (what did you do yesterday, what today and any blockers?). Scrum style.

How do you do a daily meetings when you're remote? Conference calls are the most common, but don't let anyone local dial-in or they will get in the habit of doing it. Video is best, but it's often expensive to setup. One good strategy for daily meetings is IM. The beauty of this is that there's a transcript of the daily meeting that others can read. Conference calls are usually bad b/c you can only hear the guys that are close to the phone. It's even worse when the speaker phone is a cell phone. Whatever you do, you should never skip the daily meetings.

Peer Code Reviews are another good way to get people interacting. The first rule of code reviews is that anyone can refuse to do one. If you're not doing code reviews or pair programming, you'll end up with inefficiencies and broken windows. Address the behavior head-on. Peer Code Reviews remotely: e-mail, Skype headset, IM, VNC or SubEthaEdit (works awesome). NetBeans (collaboration module) and Eclipse (communication framework) have plugins that allow SubEthaEdit-style functionality.

How do you keep software running? Continuous integration and automated testing. Keep a list of atomic features. Prioritize. Don't let people work for 2 months. All tasks should be broken down into small chunks. Tests are a good way to communicate. Don't tell me you're done because you think you're done. Show me a test that proves you're done. Don't allow broken windows. Broken windows give the impression that no one cares and more bugs happen because of this. Continuous integration is essential for remote teams.

What's the best to get fast feedback? Fast compiles, fast tests, running CI 24/7. Another good strategy is the The List. Make sure and get feedback from your Tech Lead and from your Customers (they like to be involved). If you have a list that doesn't change after 6 weeks, you're working for a team who's list is dead.

There's nothing wrong with documentation and a plan, but its the things on the left that are more important. Offsite teams should have a single tech lead. Writing Tests might be better than writing requirements. When the tests pass, you'll get what you asked for. If you care about the design, write unit tests. If you don't care about how it looks, but want it to work, use integration/MCT for functional tests. Don't separate development and QA. To deal with time zones, make the tech lead stay up and rotate. For documentation, DRY out your Docs. Generate documentation instead of manually creating them.

"Docs are a cache ... flush often"

For Customer Feedback, have the tech lead translate.

I really enjoyed Jared's talks. The first one (on Ship it!) didn't provide much new information for me, but that's likely because I've worked on several open source projects and I've tried to bring those procedures and practices into any company I've worked at. I'm very grateful that LinkedIn does a good job with continuous integration (Hudson), bug tracking (JIRA) and documentation/planning (Confluence).

The 2nd talk on Distributed Teams was enjoyable because I'm in the heart of that - managing a remote team in Denver. We've experienced the awful conference calls where you can't hear anything or when we're on speaker on a cell phone. We do have all the video conferencing equipment purchased (10K worth) and sitting in our office, but haven't set it up because we're waiting on IT to setup a persistent VPN (which is a requirement to make it work). I've had some communications issues with my boss (mostly related to him not knowing what we're working on). We do daily stand-ups and use many of the practices recommended by Jared. One thing I think we can do to increase communication with headquarters is to start doing our daily meetings over IM and posting the transcript to a wiki page.

If you get a chance to hear Jared speak, I highly recommend it. He has a style that's very conversational and fun to listen to. You feel like you're a part of a self-help group rather than listening to someone preach about how you should be developing software.

What's wrong with JSF

The developers of Seam have come up with a list of major issues with JSF. I'm assuming many of these issues are fixed by Seam, but it's interesting to note how they've somewhat admitted that JSF has many flaws. Note that there's a lot of references to Struts and WebWork.

Hopefully many of these will be fixed in JSF 2. If REST support is an important feature for web frameworks, it'll be be interesting to see how the component frameworks handle it. It'd be great if they provided native support. Oh wait, then they'd be action-based frameworks.

[OSCON 2008] Web Frameworks of the Future: Flex, GWT, Grails and Rails

Below is the presentation I'm delivering at OSCON today. Unfortunately, I had to remove slides on GWT and Flex to fit w/in the 45 minute time limit. I hope to expand this presentation in the future, as well as continue to develop the side project I'm working on using these technologies.

[OSCON 2008] Google XML Pages (GXP) by Harry Heymann and Laurence Gonsalves

GXP is a templating system that Harry and Laurence developed at Google. It was original created by Laurence in late 2001 (Java run-time, compiler written in Python) as part of the AdWords rewrite. I'm attending this session because I heard from a Google employee that they were using WebWork + their proprietary templating framework for the view. My suspicion is that GXP is that framework. The presentation I'm listening to is available at the following URL:

http://docs.google.com/Present?docid=dcbpz3ck_8gphq8bdt

Google XML Pages has the following features:

• static type checking
• convenient parameter passing/modularization system
• partial markup validation
• automatic escaping of untrusted content
• encourages functional style, discourages side-effects
• internationalization support
• lightweight runtime

GXP is an open source project as of today and is available at http://gxp.googlecode.com. It's used by AdWords, AdSense, Checkout, Blogger, Analytics, Reader and many more.

HelloWorld.gxp:

<gxp:template name='com.google.sample.HelloWorld'
              xmlns:gxp='http://google.com/2001/gxp'>
Hello, World!
</gxp:template>

Output:

Hello, World!

GXP has compile-time markup validation as well as static-type checking. GXP has native data types: for text/html, text/plain, text/css and text/javascript. It supports loops, conditionals, abbreviations, internationalization (i.e. <gxp:msg>Hello, World</gxp:msg>) with placeholders. You can call GXP in the same package using <call:GXPName>. To call something outside a package, you can use <gxp:import> to import packages or classes. You can also use a qualified XML namespace to access another package.