The LinkedIn Journey Continues

As you might know, I've spent the last several months working for one of the coolest clients ever: LinkedIn. They hired me back in July 2007 and I was impressed on day one. I was originally hired to help them evaluate open source Java web frameworks and try to determine if moving from their proprietary one to an open source one would help improve developer productivity.

After looking at all the options, I recommended we look at Struts 2 and Spring MVC - primarily because they seemed to be the best frameworks for a LinkedIn-type of application. Another Engineer and I prototyped with Struts 2 for about 6 weeks and came up with a prototype that worked quite well. While our mission was successful, we found a couple issues with Struts 2 and standard JSP that might actually hurt developer productivity more than it helped.

Following this project, I worked on the New Homepage Team, which is now visible to everyone that logs onto LinkedIn. My role was minimal, but it was still a very fun project to work on. You know those widgets in the right panel? I did the initial UI and backend integration for those. All the business logic, Ajax/JavaScript, CSS, and optimization was done by other folks on the team. Shortly after this project went live in November, I started prototyping again with Spring MVC + JSP.

The reason I was asked to prototype with Spring MVC was because they were using Spring on the backend, Spring MVC in a couple other projects, and a new project was being kicked off that used Grails. Rather than add another framework (Struts 2) to the mix, they wanted to see if they could suppress any further framework proliferation.

After a month of prototyping with Spring MVC + JSP, my results weren't as good as Struts 2. With Struts 2, I was able to use OGNL to do all the things their current JSP implementation allows them to do (call methods with arguments, use statics in EL, etc.). With standard JSP, a lot of this wasn't possible. If it was - it required writing lots of tag libraries and made it more cumbersome for developers to do certain things. At the end of that project, I determined that using FreeMarker might solve these problems. I also determined that neither Struts 2 nor Spring MVC would solve the ultimate problem of developer productivity. Neither framework would allow developers to go from make-a-change-and-deploy, wait-3-minutes-to-see-change-in-browser to make-a-change, save and wait-15-seconds-to-see-change-in-browser.

I recommended that this be the ultimate goal - to get rid of the deployment cycle and to allow minimal turnaround when deploying modified classes. After that problem was solved, it's true that moving to an open source web framework would likely provide an easier-to-remember API. However, the problem with moving to a new web framework would be that everything used to construct the existing site would suddenly become legacy code.

In the end, we concluded that the best solution might be to enhance the existing framework to be more like the available open source options. This would allow existing applications to keep using their code -- and if we enhance properly -- new applications can use a simpler, less verbose API and a templating framework that's easier to understand. We can make LinkedIn's version of JSP more like standard JSP while allowing its powerful EL to remain. We can add support for JSP Tag Libraries and Tag Files.

One of the benefits of moving to an open source web framework is there's a community, documentation and books that describe the best (or most common) ways to solve problems with the framework. LinkedIn has this, but it's all in code and no one seems to have a high-level of confidence that the way that they did it is the "best" way. Developers communicate well, but all the knowledge is stuck in their heads and inboxes - there's no way for new developers to search this knowledge and figure it out on their own without asking somebody.

By adopting an open source web framework, it's possible to solve part of this problem, but I think it's still going to exist - where a few engineers know how to use the framework really well (for the specific application) and the rest don't. We determined that regardless of open source vs. proprietary framework, what was needed was a set of developers that acted as authorities on how to develop web applications at LinkedIn. A UI Frameworks Team if you will. This would be their only job and they would never get pulled from this to work on projects or complete tasks related to LinkedIn's products. Some developers mentioned that they'd been asking for this for years, and some folks had even been hired for this. However, the formulation of this group has never happened and it's obvious (now more than ever) that it'd be awesome to have them.

The UI Frameworks Team
At the end of 6 months, it seemed my work was done at LinkedIn. I liked the idea of a UI Frameworks Team and recommended they start it with the authors of the existing web framework. They agreed this was a good idea. A few days later, I was pulled into the CTO's office and he offered me the job. He offered me the challenge of building this team and told me I could do it remotely (from Denver) and hire my own people to help me with it. I gulped as I realized I'd just been offered the opportunity of a lifetime. I knew that while this might not be the best option for LinkedIn, it certainly was an excellent opportunity for me. I said I'd think about it.

In the meantime, I was given a project which you might've read about. They asked me to migrate a Rails application to Grails and try to determine if they really needed both frameworks. I spent 2 weeks coming up to speed on both and flew to Mountain View to deliver my conclusion. Here's an excerpt from an internal blog post I wrote.

As far as I know, Rails has been used at LinkedIn for well over 6 months and Grails has been used for a similar duration. Both projects that've used these technologies have enjoyed extreme success. Both projects have been fun for the developers working on them and both have improved the technologies/frameworks they're using.

Here's an interesting quote about the Rails application:

Another app you might want to look at is BumperSticker, our facebook app. Interestingly we heard through joyent that DHH (the creator of Rails) told them that BumperSticker is the biggest rails app in the world (in terms of page views) - we are closing in on 1 billion monthly page views and we have 1 million unique users per day (about 10 million installs on FB). It's a little trickier to setup in a dev environment since you need to be running on FB, but the code itself is pretty interesting since we've iterated on it a bunch of times and are making extensive use of third party libraries such as memcached.

This quote loosely translates to "We have some Rails Ninjas on staff and we've been quite successful in developing with it and making it scale".

Both platforms have allowed developers to iterate quickly and turbo-charge their productivity.

My Conclusion: Allow Both

Why?

If you have talented developers that can whip out kick-ass code with either platform, pay them and pay them well. Passion is the most important part of any job. If developers are passionate about the application they're developing and the language they're using (notice language is secondary) - they can do great things.

I know this probably isn't the answer you wanted to hear, but it's what I believe. I think both frameworks are very similar. I believe the knowledge you gain from learning one framework is transferable to the other. A lot of the things I learned about Rails worked with Grails. Ruby's syntax is similar to Groovy's.

There's a natural synergy between these two frameworks. The hard part is figuring out when to use which one.

The application that I was asked to port from Rails to Grails? The one that was launched last week - LinkedIn Mobile.

After doing this research, I stepped up to the plate and accepted the offer to start a UI Frameworks Team and recruited some kick-ass Java Developers I know to be the founding members. Last week, I flew out to Mountain View to do some kickoff meetings and start getting the infrastructure in place so we can document, support and release code like a well-oiled open source project. There's nothing saying we won't use an open source web framework as the underlying engine, but I think this should be an excellent chance to see the power of open source governance and development style in a corporate environment.

Director of Engineering, Core Experience
I should mention one last thing. If you're an experienced Java Developer/Architect with a passion and deep knowledge of UI development (JavaScript, CSS, HTML), we've got a Director of Engineering, Core Experience position with your name on it. I might even get to interview you if you apply for this job. Furthermore, whoever gets hired will likely work very closely with my team. What's not to like about that!? ;-)

Posted in Java at Mar 06 2008, 08:00:49 AM MST 19 Comments
Comments:

congrats!

Posted by mettamara on March 06, 2008 at 09:42 AM MST #

Congrats Matt! I hope you keep posting on your Grails experiences.

Posted by Andres Almiray on March 06, 2008 at 11:36 AM MST #

Congrats Matt!

Posted by Mike Jennings on March 06, 2008 at 12:03 PM MST #

>> My Conclusion: Allow Both

Perfect!

I love this post.

Congrats buddy. You are a great asset to Linkedin. After you stepped in Linkedin is on the way up 1000 times and more.

Posted by Muthu Ramadoss on March 06, 2008 at 01:26 PM MST #

Congratulations, Matt & LinkedIn. I must spend at least a third of my life nowadays angsting out over the state of Java web UI development, so I look forward to hearing more reports from you and your team.

Posted by Ray Davis on March 06, 2008 at 01:43 PM MST #

Sounds like a nice gig.

Re: "I recommended that this be the ultimate goal - to get rid of the deployment cycle and to allow minimal turnaround when deploying modified classes."

I'd be very interesting in hearing why something like http://nicklothian.com/blog/2008/01/24/run-your-web-code-from-your-ide/ doesn't work for you. It's made a big difference here, and it means your maximum turnaround time is the time to start a debug session in your IDE. In a lot of cases you can run it in debug mode and you get the benefit of hotswapping classes, too.

Posted by Nick Lothian on March 06, 2008 at 04:42 PM MST #

Matt, thanks for the great post.

I'm curious if you have any thoughts on folks that might be trying to make a decision between Rails and Grails. I like the concept of "Allow Both", but what if you "have neither"?

If you were starting a new project, could choose either one, needed to interact with a lot of existing Java code (JRuby on Rails I guess), what would you pick?

Thanks for all your great work.

Posted by Jared Peterson on March 06, 2008 at 09:48 PM MST #

Congrats mate,

It will be good to have you flying out to Mountain View still :)

Over time, it will be interesting to see how Grails and Rails progress for your teams.

Or, maybe the next team will use Django, or Jaxer, or ... ;)

Cheers,

Dion

Posted by Dion Almaer on March 06, 2008 at 11:27 PM MST #

Matt,

This might be too late for you, but I believe you could get the developer productivity you're looking for in Spring MVC (or other existing frameworks) by using OSGi. This allows individual modules to be installed, updated and uninstalled at runtime, almost never requiring a full server restart. Spring offers comprehensive support for OSGi through the Spring Dynamic Modules (DM) framework.

Indeed, most open source and proprietary application servers are moving towards OSGi, so this technology will certainly have an influence in the future.

Regards,
Neil

Posted by Neil Bartlett on March 07, 2008 at 05:36 AM MST #

Neil - OSGi is very much on our radar and we may use Spring MVC under-the-covers as the engine for the current web framework.

Posted by Matt Raible on March 07, 2008 at 08:01 AM MST #

Congratulations Matt! This sounds like a great opportunity for you.

-Patrick

Posted by Patrick Peralta on March 07, 2008 at 08:45 AM MST #

hi Matt

Congrats on your new challenge! you definitely have a dream job at LinkedIn!
good luck
Patria

Posted by patria lukman on March 07, 2008 at 09:03 AM MST #

Congrats Matt,

Looking forward to hearing about what you choose to do with the UI Frameworks Team.

Posted by Solomon on March 07, 2008 at 09:32 AM MST #

Neither framework would allow developers to go from make-a-change-and-deploy, wait-3-minutes-to-see-change-in-browser to make-a-change, save and wait-15-seconds-to-see-change-in-browser.
Just out of interest, did they try JavaRebel? Would that help their productivity or was it some XML-based code generating blah-blah-blah framework?

Posted by Jevgeni Kabanov on March 07, 2008 at 02:02 PM MST #

Jevgeni - it's definitely possible that JavaRebel will help. I plan on experimenting with it in the next couple of weeks.

Posted by Matt Raible on March 07, 2008 at 04:33 PM MST #

I am studying about rails. What is the equivalent of ejb beans in rails? Does it matter?

Also rails is based on request model and not component model like JSF so you can't reuse GUI components well? How much that affects the decision?

I think JRuby can let users to focus on rails developmental resources and sparingly use java where it is not possible to use ruby. This way you don't have to focus on two frameworks.

..smk

Posted by smk on March 08, 2008 at 08:22 AM MST #

Matt, I truly enjoy your postings, the information you share, the experiences, your general journey through the technology landscape.. Though due to my past experiences I have mixed feelings about a framework division in a group whose primary goal is applications development. Would be keen to hear more how this goes...

Posted by Minority Report on March 09, 2008 at 06:06 PM MDT #

Gday from Australia.

Been reading your blog for sometime, a couple of years actually and i really enjoy the insight into your java career and interesting subjects you discuss.

Good to see you have hooked up with a great job. I have built up a profile on linkedin and i am very impressed with the system so far. They look like a very inovative company and hopefully they provide you with some good meaty challenges :)

Posted by Mark Wolfe on March 13, 2008 at 03:18 PM MDT #

Congrats Matt,

you surely deserve such great job and I'm happy for you.

Posted by Nicola Piccinini on March 16, 2008 at 04:09 PM MDT #

Post a Comment:
  • HTML Syntax: Allowed