Raible Designs

Free Maven Training in New Orleans on Election Day

Last week, I attended a talk by Jason van Zyl on Comprehensive Project Intelligence. This talk contained a good overview of some Maven goodies that Jason's company has been working on - namely better repository management (Nexus), better Eclipse support (m2eclipse) and improving an awesome CI server (we use Hudson heavily at LinkedIn).

Photo by Toshio

It looks like Jason is going to be doing a similar session in a full-day format at ApacheCon next week. The good news is it's free, the bad news is you have to be in New Orleans. Hopefully Jason and team will post their slides from this event and we can all learn how to use Maven more effectively.

In other Maven-related news, I hope to complete AppFuse 2.1 around the same time as Spring 3.0 (early next year). As part of that release, I hope to start using Nexus for AppFuse's repository as well as restructure its archetypes to allow using m2eclipse (with WTP) more effectively.

Colorado Software Summit 2008 Wrapup

Last week, I attended the Colorado Software Summit in Keystone and had a great time. Not only was the weather beautiful and the food delicious, the sessions I attended were awesome. If you ever get a chance to go to this conference, I highly recommend it. It's like being on vacation and learning with a bunch of old friends.

Yan Pujante also attended this conference and documents his experience, photography and presentations in Colorado Software Summit 2008.

Below is a list of my entries for all the sessions I attended.

• Building LinkedIn's Next Generation Architecture with OSGi by Yan Pujante
• Comprehensive Project Intelligence with Jason van Zyl
• What's Coming in Spring 3.0
• Applying Flash to Java: Flex and OpenLaszlo with Dustin Marx
• Building Rich Applications with Appcelerator
• Taking Apache Camel for a Ride with Bruce Snyder
• Core Animation with Bill Dudney
• RESTful Web Applications with Subbu Allamaraju

For next year, I think the conference should shorten its sessions (from 90 to 60 minutes), invite more speakers and cut the price in half (to $999 per person). How do you think the Software Summit could be improved?

Building LinkedIn's Next Generation Architecture with OSGi by Yan Pujante

This week, I'm attending the Colorado Software Summit in Keystone, Colorado. Below are my notes from an OSGi at LinkedIn presentation I attended by Yan Pujante.

LinkedIn was created in March of 2003. Today there's close to 30M members. In the first 6 months, there was 60,000 members that signed up. Now, 1 million sign up every 2-3 weeks. LinkedIn is profitable with 5 revenue lines and there's 400 employees in Mountain View.

Technologies: 2 datacenters (~600 machines). SOA with Java, Tomcat, Spring Framework, Oracle, MySQL, Servlets, JSP, Cloud/Graph, OSGi. Development is done on Mac OS X, production is on Solaris.

The biggest challenges for LinkedIn are:

• Growing Engineering Team on a monolithic code base (it's modular, but only one source tree)
• Growing Product Team wanting more and more features faster
• Growing Operations Team deploying more and more servers

Front-end has many BL services in one webapp (in Tomcat). The backend is many wars in 5 containers (in Jetty) with 1 service per WAR. Production and Development environments are very different. Total services in backend is close to 100, front-end has 30-40.

Container Challenges
1 WAR with N services does not scale for developers (conflicts, monolithic). N wars with 1 service does not scale for containers (no shared JARs). You can add containers, but there's only 12GB of RAM available.

Upgrading back-end service to new version requires downtime (hardware load-balancer does not account for version). Upgrading front-end service to new version requires redeploy. Adding new backend services is also painful because there's lots of configuration (load-balancer, IPs, etc.).

Is there a solution to all these issues? Yan believes that OSGi is a good solution. OSGi stands for Open Services Gateway initiative. Today that term doesn't really mean anything. Today it's a spec with several implementations: Equinox (Eclipse), Knoplerfish and Felix (Apache).

OSGi has some really, really good features. These include smart class loading (multiple versions of JARs is OK), it's highly dynamic (deploy/undeploy built-in), it has a service registry and is highly extensible/configurable. An OSGi bundle is simply a JAR file with an OSGi manifest.

In LinkedIn's current architecture, services are exported with Spring/RPC and services in same WAR can see each other. The problem with this architecture comes to light when you want to move services to a 2nd web container. You cannot share JARs and can't talk directly to the other web app. With OSGi, the bundles (JARs) are shared, services are shared and bundles can be dynamically replaced. OSGi solves the container challenge.

One thing missing from OSGi is allowing services to live on multiple containers. To solve this, LinkedIn has developed a way to have Distributed OSGi. Multicast is used to see what's going on in other containers. Remote servers use the OSGi lifecycle and create dynamic proxies to export services using Spring RPC over HTTP. Then this service is registered in the service registry on the local server.

With Distributed OSGi, there's no more N-1 / 1-N problem. Libraries and services can be shared in one container (memory footprint is much smaller). Services can be shared across containers. The location of the services is transparent to the clients. There's no more configuration to change when adding/removing/moving services. This architecture allows the software to be the load balancer instead of using a hardware load balancer.

Unfortunately, everything is not perfect. OSGi has quite a few problems. OSGi is great, but the tooling is not quite there yet. Not every library is a bundle and many JARs doesn't have OSGi manifests. OSGi was designed for embedded devices and using it for the server-side is very recent (but very active).

OSGi is pretty low-level, but there is some work being done to hide the complexity. Spring DM helps, as do vendor containers. SpringSource has Spring dm Server, Sun has GlassFish, and Paremus has Infiniflow. OSGi is container centric, but next version will add distributed OSGi, but will have no support for load-balancing.

Another big OSGi issue is version management. If you specify version=1.0.0, it means [1.0.0, ∞]. You should at least use version=[1.0.0,2.0.0]. When using OSGi, you have to be careful and follow something similar to Apache's APR Project Versioning Guidelines so that you can easily identify release compatibility.

At LinkedIn, the OSGi implementation is progressing, but there's still a lot of work to do. First of all, a bundle repository needs to be created. Ivy and bnd is used to generate bundles. Containers are being evaluated and Infiniflow is most likely the one that will be used. LinkedIn Spring (an enhanced version of Spring) and SCA will be used to deploy composites. Work on load-balancing and distribution is in progress as is work on tooling and build integration (Sigil from Paremus).

In conclusion, LinkedIn will definitely use OSGi but they'll do their best to hide the complexity from the build system and from developers. For more information on OSGi at LinkedIn, stay tuned to the LinkedIn Engineering Blog. Yan has promised to blog about some of the challenges LinkedIn has faced and how he's fixed them.

Update: Yan has posted his presentation on the LinkedIn Engineering Blog.

[DJUG] Ship it! and Distributed Teams by Jared Richardson

Jared's first book is called Ship it! and it's been wildly successful (translated into several different languages). During the first hour, Jared talked about his book and practices defined in it.

Ship it! is an attempt to talk about the holistic process of software development: techniques, infrastructure and process. For Techniques, there's a Technical Lead, The List, Code Reviews, Code Change Notifier and Daily Meetings. Infrastructure involves Version Control, Script Builds, Continuous Builds, Track Issues, Track Features and Writing/Running Tests. Process is Propose System Objects -> Propose Interfaces -> Connect Interfaces -> Add Functions -> Refactor, Refine, Repeat.

Martin Fowler's definition of Architecture: anything that is difficult to change later.

The majority of Jared's talk was about implementing TDD, continuous integration and many other things driven by audience questions. I didn't take notes because it was more of a conversation than a presentation. Jared's second talk was on Distributed Teams and my notes from this talk are below.

Agile is not XP or Scrum, those are implementations of Agile. The Manifesto for Agile Software Development is the interface for Agile Development. It contains the following principles:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Custom collaboration over contract negotiation
• Responding to change over following a plan

To re-word these principles: Agility is...

• People interacting
• Software that runs
• Fast Feedback
• Frequent Change

Scrum principles are often not about developing software, they're about making people talk. The important thing is to get people in the habit of communicating. The best way to get people interacting is to do Daily meetings. One to two minutes per person. Same time. Three questions (what did you do yesterday, what today and any blockers?). Scrum style.

How do you do a daily meetings when you're remote? Conference calls are the most common, but don't let anyone local dial-in or they will get in the habit of doing it. Video is best, but it's often expensive to setup. One good strategy for daily meetings is IM. The beauty of this is that there's a transcript of the daily meeting that others can read. Conference calls are usually bad b/c you can only hear the guys that are close to the phone. It's even worse when the speaker phone is a cell phone. Whatever you do, you should never skip the daily meetings.

Peer Code Reviews are another good way to get people interacting. The first rule of code reviews is that anyone can refuse to do one. If you're not doing code reviews or pair programming, you'll end up with inefficiencies and broken windows. Address the behavior head-on. Peer Code Reviews remotely: e-mail, Skype headset, IM, VNC or SubEthaEdit (works awesome). NetBeans (collaboration module) and Eclipse (communication framework) have plugins that allow SubEthaEdit-style functionality.

How do you keep software running? Continuous integration and automated testing. Keep a list of atomic features. Prioritize. Don't let people work for 2 months. All tasks should be broken down into small chunks. Tests are a good way to communicate. Don't tell me you're done because you think you're done. Show me a test that proves you're done. Don't allow broken windows. Broken windows give the impression that no one cares and more bugs happen because of this. Continuous integration is essential for remote teams.

What's the best to get fast feedback? Fast compiles, fast tests, running CI 24/7. Another good strategy is the The List. Make sure and get feedback from your Tech Lead and from your Customers (they like to be involved). If you have a list that doesn't change after 6 weeks, you're working for a team who's list is dead.

There's nothing wrong with documentation and a plan, but its the things on the left that are more important. Offsite teams should have a single tech lead. Writing Tests might be better than writing requirements. When the tests pass, you'll get what you asked for. If you care about the design, write unit tests. If you don't care about how it looks, but want it to work, use integration/MCT for functional tests. Don't separate development and QA. To deal with time zones, make the tech lead stay up and rotate. For documentation, DRY out your Docs. Generate documentation instead of manually creating them.

"Docs are a cache ... flush often"

For Customer Feedback, have the tech lead translate.

I really enjoyed Jared's talks. The first one (on Ship it!) didn't provide much new information for me, but that's likely because I've worked on several open source projects and I've tried to bring those procedures and practices into any company I've worked at. I'm very grateful that LinkedIn does a good job with continuous integration (Hudson), bug tracking (JIRA) and documentation/planning (Confluence).

The 2nd talk on Distributed Teams was enjoyable because I'm in the heart of that - managing a remote team in Denver. We've experienced the awful conference calls where you can't hear anything or when we're on speaker on a cell phone. We do have all the video conferencing equipment purchased (10K worth) and sitting in our office, but haven't set it up because we're waiting on IT to setup a persistent VPN (which is a requirement to make it work). I've had some communications issues with my boss (mostly related to him not knowing what we're working on). We do daily stand-ups and use many of the practices recommended by Jared. One thing I think we can do to increase communication with headquarters is to start doing our daily meetings over IM and posting the transcript to a wiki page.

If you get a chance to hear Jared speak, I highly recommend it. He has a style that's very conversational and fun to listen to. You feel like you're a part of a self-help group rather than listening to someone preach about how you should be developing software.

EhCache Project Busy this Summer

The EhCache project appears to be having a very busy summer. EhCache 1.5.0 (a major new version) was released on July 12th. In addition, a new (SOAP-based) EhCache Server was released at the end of July. You might ask yourself why you'd need such a beast. I think Greg explains it best:

Why am I doing this? There are lots of theories that have made their way on to the ehcache mailing list. The prosaic truth is that a large US corporate using ehcache for their Java apps on 200+ servers also wants to use it for their C++ apps. And they are prepared to sponsor development. The Web Services API lets them do it. That's it.

As to the larger question of how interesting this is to the world at large, my view is not very. However having to jump through all the hoops to get a server infrastructure done, I thought that the world at large may be interested in a RESTful, resource oriented ehcache server.

The next day, Greg announces EhCache for JRuby on Rails. A few days later, RESTful, resource-oriented caching becomes available in ehcache-server.

I guess this helps answer the question about OSCache vs. EhCache. OSCache hasn't had a release in over a year and EhCache is pumping out new releases and new products. Well done, Greg!

At LinkedIn, we use EhCache for many of our caching needs. However, it's likely we'll be moving to Memcached in the future. Since I'll be part of the team that implements Memcached, it'll be interesting to see which one performs better.

LinkedIn Tech Talk: Kevin Brown on Shindig

Last Thursday, Kevin Brown visited LinkedIn's Mountain View office to do a presentation on Shindig, an OpenSocial Reference Implementation. Below are my notes from his talk.

In September 2007, Google started thinking about Social APIs. Google Gadgets would be better with access to Social Data ... but that's just Google. It was recognized that this is something that many others would like access to. OpenSocial was announced in November 2007. It's an open standard for developer platforms that has a strong emphasis on "social" data. It's based on gadgets which is now covered by The Open Social Foundation.

In November, many Googlers started working on a Google Code project based on Java and iGoogle. However, there was too much proprietary code. In December, Brian McCallister of Ning created an ASF Proposal for Shindig. It was a rough port of iGoogle but with Ning's PHP code. This turned out to be a great starting point. It immediately got interest from Google, Hi5, MySpace and others. While most committers are still from Google, there are 12 developers that work on it full time and they're adding 2 committers each month. Shindig is a Java/PHP implementation of OpenSocial. Open Campfire is an Apache-licensed .NET implementation that hopes to eventually merge into Shindig.

Read more on the LinkedIn Blog »

LinkedIn has the Biggest Rails app in the World

From the LinkedIn Engineering Blog:

Bumper Sticker started as a small experiment in August, 2007. Facebook had released their development platform while we were hard at work on our own. We were curious to experiment and discover some of the characteristics of an application platform built on a social network and to see what, if any, learning we could apply to our own efforts. After noticing that professional and business-related applications weren't flourishing in the Facebook ecosystem, a few of our Product folks put their heads together while out for a run; one engineer, one week, and a few Joyent accelerators later, Bumper Sticker was born.

We'd be lying if we said that anyone was prepared for the kind of success Bumper Sticker has had since then - though we should have expected it, given the excellent Product team here at LinkedIn. Here's a quick snapshot of Bumper Sticker statistics at this moment: Read More »

The "biggest Rails app in the world" claim comes from this video.

In addition to having a kick-ass RoR team at LinkedIn, we also do a lot with Java and love our Macs. Why wouldn't you want to work here?

If you find a gig you like, or simply have mad programming skills, contact me and I'll see if I can hook you up. And yes, we are hiring at LinkedIn Denver.

Is it possible to replace the syntax parser in Eclipse or IDEA's JSP Plugin?

At LinkedIn, we have our own JSP Compiler. Our version of JSP is more like FreeMarker than JSP since it solves many of the deficiencies of JSP. Since we allow a different syntax than standard JSP (more powerful EL, new tags for looping, loading from classpath), we (like FreeMarker) don't get much love from IDEs.

We don't get much in the way of syntax-highlighting or code completion. However, since we use JavaCC/JJTree for parsing, I'm wondering if Eclipse or IDEA (or even NetBeans) allows replacing the default syntax definition with a new one.

Has anyone extended one of these IDEs to enhance its JSP syntax highlighting and compilation? If so, I'd love to hear about it. If not, it's likely we'll be doing it in the near future.

LinkedIn's Engineering Blog

Have you been curious about LinkedIn's architecture or how they're using Grails and Rails? If so, you might be interested in LinkedIn's Engineering Blog. Over the past couple of weeks, a few Engineers have starting writing about our architecture, OpenSocial, RailsConf, YUI, Grails and OSGi. Below is a complete listing of Engineering posts.

• OSGi at LinkedIn: Java Compilation in OSGi
• Grails at LinkedIn
• OSGi at LinkedIn
• LinkedIn Speaks YUI
• Enlightenments from RailsConf 2008
• Google I/O and LinkedIn's Open Social Integration
• LinkedIn at JavaOne 2008

If there are topics you'd like to see us blog about, please let me know. I've somehow landed in the role of Editor for the Engineering Blog, so I should be able to hook you up if I can find an engineer to blog about what you're interested in.

On a related note, Rob Getzschman's entry LinkedIn discovers the truth about Cannes is quite entertaining. Highly recommended.

RE: What's a good RIA to develop in 20 hours?

Thanks to everyone who commented on my previous post and offered recommendations for RIAs to develop in 20 hours or less. In order to narrow down my choices, I've created a survey on SurveyMonkey.com. Here's a list of the application ideas I received from comments and e-mails:

1. Lightweight CMS
2. MP3 Player
3. Resume Editor/Publisher
4. Meal/Calorie Tracker
5. Contact Management
6. Planning Application
7. Timesheet Application
8. DB/SQL Client
9. Status Updater/Aggregator (LinkedIn, Twitter and Facebook)
10. Online File Explorer (browser-based FTP interface)

I like #3 (Resume) and #9 (Status) because I may be able to tie those into LinkedIn's RESTful API.

Click here to vote for the application you'd like me to develop »

Voting ends at noon on Friday (Mountain Time).

Update: My co-workers had a good suggestion at lunch today: pitchersacrossamerica.com. It seems it's kinda difficult to find bars that serve pitchers these days (at least in Denver). Create an app that allows people to enter in bars and restaurants that serve pitchers and show them on a map. Seems simple and fun. If enough people like the idea, I'll restart the survey with this as an option. In the meantime, the current (Wednesday night) numbers are here.

Update 2: Here's the results as of Thursday night. Only 15.5 more hours to vote!

Update 3: Final Results. Thanks to everyone who voted! I'm traveling a lot next week (Mountain View followed by Boston), but I'll try to write an entry on next steps.