Raible Designs

The LinkedIn Journey Continues

As you might know, I've spent the last several months working for one of the coolest clients ever: LinkedIn. They hired me back in July 2007 and I was impressed on day one. I was originally hired to help them evaluate open source Java web frameworks and try to determine if moving from their proprietary one to an open source one would help improve developer productivity.

After looking at all the options, I recommended we look at Struts 2 and Spring MVC - primarily because they seemed to be the best frameworks for a LinkedIn-type of application. Another Engineer and I prototyped with Struts 2 for about 6 weeks and came up with a prototype that worked quite well. While our mission was successful, we found a couple issues with Struts 2 and standard JSP that might actually hurt developer productivity more than it helped.

Following this project, I worked on the New Homepage Team, which is now visible to everyone that logs onto LinkedIn. My role was minimal, but it was still a very fun project to work on. You know those widgets in the right panel? I did the initial UI and backend integration for those. All the business logic, Ajax/JavaScript, CSS, and optimization was done by other folks on the team. Shortly after this project went live in November, I started prototyping again with Spring MVC + JSP.

The reason I was asked to prototype with Spring MVC was because they were using Spring on the backend, Spring MVC in a couple other projects, and a new project was being kicked off that used Grails. Rather than add another framework (Struts 2) to the mix, they wanted to see if they could suppress any further framework proliferation.

After a month of prototyping with Spring MVC + JSP, my results weren't as good as Struts 2. With Struts 2, I was able to use OGNL to do all the things their current JSP implementation allows them to do (call methods with arguments, use statics in EL, etc.). With standard JSP, a lot of this wasn't possible. If it was - it required writing lots of tag libraries and made it more cumbersome for developers to do certain things. At the end of that project, I determined that using FreeMarker might solve these problems. I also determined that neither Struts 2 nor Spring MVC would solve the ultimate problem of developer productivity. Neither framework would allow developers to go from make-a-change-and-deploy, wait-3-minutes-to-see-change-in-browser to make-a-change, save and wait-15-seconds-to-see-change-in-browser.

I recommended that this be the ultimate goal - to get rid of the deployment cycle and to allow minimal turnaround when deploying modified classes. After that problem was solved, it's true that moving to an open source web framework would likely provide an easier-to-remember API. However, the problem with moving to a new web framework would be that everything used to construct the existing site would suddenly become legacy code.

In the end, we concluded that the best solution might be to enhance the existing framework to be more like the available open source options. This would allow existing applications to keep using their code -- and if we enhance properly -- new applications can use a simpler, less verbose API and a templating framework that's easier to understand. We can make LinkedIn's version of JSP more like standard JSP while allowing its powerful EL to remain. We can add support for JSP Tag Libraries and Tag Files.

One of the benefits of moving to an open source web framework is there's a community, documentation and books that describe the best (or most common) ways to solve problems with the framework. LinkedIn has this, but it's all in code and no one seems to have a high-level of confidence that the way that they did it is the "best" way. Developers communicate well, but all the knowledge is stuck in their heads and inboxes - there's no way for new developers to search this knowledge and figure it out on their own without asking somebody.

By adopting an open source web framework, it's possible to solve part of this problem, but I think it's still going to exist - where a few engineers know how to use the framework really well (for the specific application) and the rest don't. We determined that regardless of open source vs. proprietary framework, what was needed was a set of developers that acted as authorities on how to develop web applications at LinkedIn. A UI Frameworks Team if you will. This would be their only job and they would never get pulled from this to work on projects or complete tasks related to LinkedIn's products. Some developers mentioned that they'd been asking for this for years, and some folks had even been hired for this. However, the formulation of this group has never happened and it's obvious (now more than ever) that it'd be awesome to have them.

The UI Frameworks Team
At the end of 6 months, it seemed my work was done at LinkedIn. I liked the idea of a UI Frameworks Team and recommended they start it with the authors of the existing web framework. They agreed this was a good idea. A few days later, I was pulled into the CTO's office and he offered me the job. He offered me the challenge of building this team and told me I could do it remotely (from Denver) and hire my own people to help me with it. I gulped as I realized I'd just been offered the opportunity of a lifetime. I knew that while this might not be the best option for LinkedIn, it certainly was an excellent opportunity for me. I said I'd think about it.

In the meantime, I was given a project which you might've read about. They asked me to migrate a Rails application to Grails and try to determine if they really needed both frameworks. I spent 2 weeks coming up to speed on both and flew to Mountain View to deliver my conclusion. Here's an excerpt from an internal blog post I wrote.

The application that I was asked to port from Rails to Grails? The one that was launched last week - LinkedIn Mobile.

After doing this research, I stepped up to the plate and accepted the offer to start a UI Frameworks Team and recruited some kick-ass Java Developers I know to be the founding members. Last week, I flew out to Mountain View to do some kickoff meetings and start getting the infrastructure in place so we can document, support and release code like a well-oiled open source project. There's nothing saying we won't use an open source web framework as the underlying engine, but I think this should be an excellent chance to see the power of open source governance and development style in a corporate environment.

Director of Engineering, Core Experience
I should mention one last thing. If you're an experienced Java Developer/Architect with a passion and deep knowledge of UI development (JavaScript, CSS, HTML), we've got a Director of Engineering, Core Experience position with your name on it. I might even get to interview you if you apply for this job. Furthermore, whoever gets hired will likely work very closely with my team. What's not to like about that!?

Jack's got a bead stuck in his nose!

This morning around 9:45 AM, I got a call from Jack's Teacher. She said, "Jack's got a bead stuck in his nose!" I heard a screaming kid in the background, so the first thing I asked was - "Is that Jack?" She said yes, and I was out the door a few seconds later.

Several minutes later, I arrived at his school and picked him up to take him to the Emergency Room. His teacher handed me the bead you see in the picture on the right. This was a replica of the one he jammed into his nose. I figured the ER was the best place to go considering all I saw was blood when I looked in his nostril.

10 minutes later we were in the ER and within a half hour we were talking to a nurse. Julie arrived just as the doctor walked up to talk to Jack. The first thing he suggested was that Julie hold one nostril and give him a CPR-type breath/blow into his mouth. She tipped his head back, plugged the free nostril and "pop!" - it came right out!

I wish I'd known that trick early this morning. How cool would it have been to walk into Jack's class, grab him by the head and blow that sucker out like I knew what I was doing? The good news is now you have this knowledge and you can be the hero in your kid's class someday.

Last Night's Selenium Users Meetup at Google

Last night, I attended the inaugural Selenium User Group meetup at Google's Campus in Mountain View. It was an excellent event, with many of the core committers on hand to present and answer questions. Each presenter had about 5 minutes to speak and we learned many things about the Selenium Project itself, what's coming in the future and how Google has standardized on Selenium as their integration testing tool of choice.

Patrick Lightbody started the meeting by going over a number of project statistics with pretty graphs and such. There were simply too many numbers to write down, so hopefully his slides will be published soon. I was pleased to see that Google did videotape the entire event, so it should be available online soon. I'll update this post when it it. Below are my notes from the event.

Jason Huggins, Test Engineer at Google
Selenium is a test automation framework. Some folks abuse it as a macro tool. There's two reasons Selenium became so popular: it was able to test Ajax before any other testing tool and it allows end-to-end workflow testing. Selenium works on any platform, with any browser and allows many, many languages. It's possible that other frameworks that are more focused are better. There are 4 Selenium products:

• Selenium Core (TestRunner)
• Selenium IDE (for Firefox)
• Selenium Remote Control
• Selenium Grid

Paul Hammant, Open Source Manager at Thoughtworks
Selenium 1.0 - beta this week. RC, Core, Grid and IDE together. 1.0 will be shipped in a few weeks. Compared to today, lots of bugs killed, documentation improved and a greatly improved Selenium IDE.

Selenium 1.1 will be shipped in a couple months. Selenium IDE will be enhanced to obey RC instruction ... becoming the best mode of operation when it ships.

Some experimental iPhone-Safari capability (dependent on SDK for iPhone).

Selenium 2.0 will hopefully be released this year. Roll in of WebDriver functionality and code. New model-based API. IE plugin and Safari plugin (not really a plugin, most likely uses AppleScript).

Philippe Hanrigou (http://ph7spot.com)
Testing is good! Selenium is awesome! However, end-to-end web testing is slow. It is and always will be slow. How do you solve this problem? You solve it the same way we've solved slow traffic in the past - by building more lanes. Rather than one browser testing everything, Selenium Grid allows multiple lanes and 20 browsers. Features include:

• Faster Builds, Faster Feedback!
• Easy Install and Everyday Use
• No Need to Change Your Tests
• Leverage Your Existing Computing Infrastructure

One of the best parts about Selenium Grid is you can download and have it running in 10 minutes or less. Selenium Grid comes out-of-the-box with Amazon EC2 support.

Jennifer Bevan, Lead from Selenium Farm Project at Google
As of Friday, Google has over 50 teams running over 51K tests per day on internal Selenium Farm. 96% of these tests are handled by Selenium RC and the Farm machines correctly. The other 4% are partly due to RC bugs, partly to test errors, but isolating the cause can be difficult. Selenium has been adopted as the primary technology for functional testing of web applications within Google. That's the good news.

The bad news is Google is pushing the limits of Selenium. Using Selenium (RC + Core) at this scale exposed certain issues not originally anticipated as high-impact. IE/XPath slowness has a high impaction given that can only run one test at a time. Tests can cause many conditions from which RC cannot easily or automatically recover (for example, tests that don't call stop() in every exit path). Unexpected browser dialogs, popups, etc. eventually cause timeout exceptions.

Googlers expect that RC will work for the most part, but they want it to be more reliable, with better performance. So they have:

1. Created utility methods to improve performance when examinging large tables, overlaying domain-specific languages, etc.
2. Deployed retry policies based on failure reason.

Looking to the future - Google has not yet reached their expected usage of Selenium RC. Some projects cannot use the Farm until RC supports session-level configuration (not server-level). Many just want RC to be more reliable. So Google will:

1. Continue to contribute to RC, Core and to user-created helper libraries.
2. Keep doing so until all failed tests are not Selenium's fault.

Dave Astels, Google (Driving Selenium with RSpec)
Using RSpec, you can create a very easy-to-read Story with Scenarios that can be read (and likely written) by practically anyone. Dave then uses a small script to load up the stories and run them in Selenium. When he runs the script, the scenario is spit out and test pass/fail information. Learn more at rspec.info.

Alex Chaffee, Mad Scientist at Pivotal Labs (The Selenium/Ruby Project that Must Not Be Named)
What is Polonium? It's also known as Selenium RC Fu or Selenium On Rails 2 or Funkytown. It has simple extensions to Selenium RC:

• wait_for
• element assertions
• launching/managing servers locally

Blackbox testing you're sitting out of the box and send in stuff. In whitebox testing, you get to open up the box and look at stuff. With Selenium, you can do Graybox testing, where you are doing blackbox testing (against the UI) and querying your database (or other resources) at the same time.

Dan Faulich, Sr. QA Engineer at Redfin Corporation (How Not to Run a Successful Open Source Project)
The first thing you don't want to do is support everything. The second thing you don't want to do is write your project in JavaScript. While it's great that almost anyone can hack on it, running in a sandbox sucks. Don't use a language that's bound by other people's security restrictions. Don't roll your own multi-language remoting. It's written using XML + XSLT and its such a pain in the ass to maintain that Dan is the only one that fixes bugs.

Shinya Kasatani, Developer of Selenium IDE
Selenium IDE is a Firefox extension that can record and play back tests in your browser. It can translate the recorded tests to many languages.

Selenium IDE 1.0 adds support for TestSuites. Another new feature is better recording features - it detects when the DOM is modified. Shinya has a demo where he uses the new IDE to test the Dojo Dijit Theme Test Page. Apparently, this doesn't work in the current version.

The Goal of Selenium IDE is to get more people interested in test automation of web applications and to help their projects to be successful.

Haw-bin Chai, Developer at CommerceHub
XPath is a powerful selection took, but it'd be great if we could use something like "article 5" instead of the cryptic //table[5]/tbody/... syntax.

Why UI-Element? Traditional locations can be ugly and they don't convey purpose. UI-Element is a Selenium Core extension and has Selenium IDE integration. It's written in 100% JavaScript. Examples:

ui=frontPage::topStoriesCountry()
ui=listingPages::article(index=5)
ui=listingPages::articleSource(articleIndex=1)

The UI Map is in JavaScript shorthand. It contains logic to map ui locators to traditional locations. Each UI element implements getLocator(), then you add a page set and add an element - all in JSON Format.

Simon Stewart on Web Driver
What on earth is Web Driver? One of the main problems with Selenium is it's written in JavaScript and it's too easy to hack. WebDriver is an idealized browser which allows you to do browser things like call get(URL), findElement(By.xpath()) and getTitle(). WebDriver is similar to Selenium RC, but it's not written using JavaScript. It's written in the native languages for each browser, which allows you to break out of the JavaScript Jail. The IE Driver is written in COM (C++). The Firefox Driver is written as a Firefox extension. Safari uses Apple events.

Soon there won't be a WebDriver project ... because it will be part of Selenium!

WebDriver likely won't be part of the core until Selenium 2.0 later this year. One of the nice things about WebDriver is you can implement different browsers. Out-of-the-box, it will support all the major browsers, as well as HtmlUnit with JavaScript turned off.

I had a great time learning more about Selenium and how most of its problems will be solved in the near future. The beers afterwards weren't so bad either.

Update: Videos of this event have been posted.

David Sachdev on Web Framework Proliferation

David Sachdev left the following comment in my post about the Java Web Framework Smackdown at TSSJS in Vegas:

Wow - that's quite a mouthful David. Well written!

P.S. The Early Bird Deadline for TSSJS is today.

Dynamic Language Shootout: Groovy vs. Jython vs. JRuby

Travis Jensen has an interesting post titled Our Dynamic Language Shootout:

...
For a variety of deployment reasons, we've decided that whatever we choose will be deployed on the JVM. As a result, this comparison is for the JVM versions of the languages, e.g. JRuby, Jython, and, of course, Groovy, which has no other deployment option. I want to also clarify that I have the most experience with Python and I really like the language. There is no doubt that the language influenced me in my evaluation, but I really tried to remain objective in spite of that.
...
As I did the evaluation, I tried to come up with a broad spectrum of important information. Others at my company gave feedback on the important characteristics. In the end, these are the features that we felt were most important: the interaction between Java and the selected language, the IDE support, the learning curve, existing web frameworks, and the existing community support for the JVM implementation of the language.

His conclusion: Groovy.

I don't think it should surprise you at this point that we chose Groovy. Even being openly biases towards Python first and Ruby second (hey, it's cooler :), I could not, in good conscience, choose either of them for melding into our existing environment.

If I were starting from scratch on a project, my choice would be very different. If I wanted to target the JVM, I would choose JRuby (at least until Jython 2.5 and Django are available); if I wasn't targeting the JVM, then it would be, for my Python, but I'd be equally comfortable choosing Ruby.

Well written Travis - I look forward to reading more about the new life you're breathing into your stilted development practices.

WebORB: Have you ever heard of it?

A colleague sent me an e-mail today and asked me if I'd ever heard of WebORB today. Since I hadn't, I figured I'd write this post and see if any of you have heard of it? If so, what is it and what does it do? It it similar to Appcelerator, but server-side only? Or is it more like Granite DS?

[Read More]

Leasons learned from using Seam

Yesterday, I noticed the Seam Developers released a new seamframework.org site. It's great to see a web framework team eating their own dog food. Of course, if all open source framework developers were paid full-time to work on their respective project, we'd likely see more of this.

My favorite part of the new site is the Forums, which has an Atom Feed you can use to monitor topics posted. This morning, I noticed a topic from Daniel Hinojosa titled ANN: amazinggates.com is alive with Seam & lessons learned. In this topic, Daniel lists a number of lessons he learned from working with Seam.

The one thing I noticed about Daniel's "Amazing Gates" site is it seems extremely fast. Do you think this is because of Seam or did he follow the rules for high performance websites?

The New Javalobby Sucks?

I didn't say it, Jesse Sightler did. Even though he didn't say "it sucks" explicitly, that's what I read in his post:

Is it just me, or has the new Javalobby proven to be a significant step backwards? The old site was a Slashdot style discussion system with a pace very appropriate to the pace of news flowing from the Java community. The light emphasis on announcements was welcome, and useful while at the same time not being overstated.

The new site feels a lot like TheServerSide.Com from a few years ago. They've gone to a system where the frontpage is updated frequently (many times per day) and the content there is seldom interesting enough to attract any significant discussion. Unfortunately, this means that the overwhelming number of articles on the frontpage appear dry and uninteresting. I don't think I've really read anything there since the switch to the new format.

For the sake of the site, I do hope they figure out their mistake here. There is no shame in turning this into worsethanfailurethedailywtf all over again (hopefully you get that reference).

I like the new site because I visit it more than the old one. Of course, that could be a direct result of me posting there. If I could change one thing, I'd like to see a java.blogs-style aggregator of all zones (then I'd turn off the .NET and Kids Code Zones).

Do you agree with Jesse? Should Javalobby change back to the old-way of using forums?

I believe the reason for the change was because DZone has become so much more popular than Javalobby. I think they're hoping to capitalize on that brand name and extend it to other communities. Look at the following graph from Alexa for proof. More traffic = more $$ from advertisers.

Web Application Frameworks based on Real-World Popularity

I received an interesting (spam?) comment on my What Web Application framework should you use? entry today:

A useful resource to compare Java web frameworks (Spring, Tapestry, Struts, OpenLaszlo,...) and also PHP, Python, Ruby web frameworks:

http://www.therightsoft.com/softwaretechnologies/webframeworks

If you go to the site, you'll see they have a hierarchical list of web application frameworks based on real-word popularity. First of all, I'm unsure of what "real-word" popularity is.

Let's assume this is a typo and it should be "real-world" popularity. Where is the credible source for this data? Where is the link to this credible source? I like the list, its sortability and filterability, but there's no evidence that it's true. Care to elaborate on your sources [email protected]?

Building a Better Maven with Ant

It looks like the Ant folks are thinking of building a better Maven.

I see many developers adopt Maven because they want a build system able to provide common features with no effort. Most of them don't want to spend much time writing an Ant script, or have seen or heard that maintaining Ant build scripts is troublesome. So they choose to use Maven only because it's easy to use for common use cases: install, write a simple pom of a few lines or generate it using an archetype, and you're ready to compile, test and package your new project following the Maven standard structure. They also get dependency management for free, and with only a few more effort they have multi module builds, and some nice features like code analysis, coverage, and a set of report gathered in a web site. That's really nice and that's what I like about Maven.

But Maven suffers from a lack of flexibility and robustness IMHO. And later the same people who first adopted Maven because of its perceived ease of use become frustrated when they need to tweek the system to their own needs or don't understand how the release plugin work. Then some of them go back to Ant, first having to go through a sometimes painful road to describe their whole build system in xml, especially if they aren't Ant experts. Others try to use new build tools like raven, buildr or others.

I really like Ant, and think it is a very good basis for robust and flexible build systems. People with enough knowledge of Ant can write very good build systems, testable, maintainable and adaptable. But you need to get your hands dirty, and you need to get a good knowledge of some of the mechanisms which can make an Ant based build system manageable: import, scripts and scriptdef, macrodef, presetdef, and so on. [Read More]

What do you think - is this a good idea?

I agree that Maven has its warts, but I don't think it's that bad. I've also heard that Maven has been successfully implemented at large companyies like eBay, Intuit and E*Trade[1]. Is the "Maven sucks" meme largely something that exists in the blogosphere, but not in the real world?

I think the biggest benefit of Maven is dependency management. I think it makes your code more modular and easier to build. Rather than having a monolithic source-code tree that depends on itself being built in a certain order, you can have individual modules that pull dependencies from a central location. This can be done with Maven's Ant Tasks as well. I don't see a problem with building a better Maven with Ant, but to try and build a better Central Repository sounds like a nightmare to me. The current repository has been improved for years and is much better than it was a couple years ago. That being said, I would love to see somebody build a more accurate Central Repository. Ideally, it'd be done sometime next week.

[1] I could be wrong about these companies. If you're a developer at one of these companies, please confirm or deny. Any comments on Maven's success at these companies would be great as well.

Update: Speaking of Maven, there's an interesting comment on a Javalobby post I wrote:

With all the critical remarks the Maven project is receiving, wouldn't it be time for some Maven project lead to step up and explain the team's position? Or is it completely deaf to the sentiments? How many builds have to fail, how much more headaches are needed before others start their own version of Maven and do it the right way (like Don [Brown])?

Seems like an excellent question to me. Guys?