Raible Designs

RESTful Web Applications with Subbu Allamaraju

Subbu works at Yahoo! developing standards, patterns and practices for HTTP web languages. In the past, he was a web service and Java developer. He was also a standards contributor at BEA and an author of books on Java EE. His current passion is HTTP and REST. Subbu confesses that he's not a web developer, has no interest in the internals of programming models used for web frameworks and he's only interested in the visible aspects of the architecture.

"The Web is Mostly Restful"

Being RESTful in an abstract sense means:

• Resources are named by URIs
• Resources have representations (Atom, HTML, JSON, XML)
• Resources contain contextual links to allow navigation of state
• There's a Uniform Interface

In the web today, some resources and URIs are personalized, but most are not. Some depend on sessions, but most do not. The consequence of a personalized UI with a non-unique URI is you cannot rely on browser caching.

The web is full of different representations (HTML, XML, JS, PDF, CSS, Flash). The problem with HTML is you can't tell links that you want a particular representation based of a link. The links are hard-coded to be a particular content-type. However, some media types on responses are ignored. This is often a problem with browsers and whether the user has plugins installed.

The Uniform Interface for the web is HTML and primarily links and forms (GET and POST). There's still some misconceptions (e.g. POST is secure). However, it's not about security, it's about idempotency and safety. You need to make sure you only use POST when you're changing data. POSTs are not repeatable. GET URIs are not always refreshable, which is quite unfortunate. Users shouldn't have to fight the back button.

Caching is a fundamental aspect of the web. Even in a personalized site, most of the content can be cached. The web is read-only for the most part. However, many enterprise web applications don't take advantage of caching. This is fine when there's not that many users, but it's bad when you want to scale to thousands of users. There's several frameworks that use cache-busting and prefer backend caching over HTTP caching. These frameworks are not using the web like they should.

Backend caching (e.g. Memcached) uses a non-uniform interface and you need to explicitly program to it. Frontend/HTTP caching has a uniform interface that's pluggable. Backend caching is generally more expensive to develop and deploy. There are cases where data should be cached on the backend, but you shouldn't focus all on backend caching w/o doing some frontend caching.

With Ajax, you get more opportunities to be RESTful. XMLHttpRequest is another HTTP client that can be programmed to. It has full support for the uniform interface, which allows content negotiation, optimistic concurrency and caching. Cross-domain hacks can be done with <script> and <iframe> to tunnel requests over GET. The W3C has been working for the last two years on how to do cross-domain Ajax w/o using hacks. The problem with current cross-domain implementations is they often use GET for everything, which isn't very RESTful. Subbu recommends using a proxy on the same domain if you do need to talk to other domains. This will allow your Ajax code to remain RESTful.

Web Frameworks
Web development is hard because of all the moving pieces that exist. Because of this, many web frameworks have been created to solve the various problems. In 1997, there were servlets. They provided basic plumbing and closely reflected HTTP/1.1. Servlets provided a poor programming model, but it allowed a lot of frameworks to be built on top of it. We don't use servlets to write applications, only to write application frameworks. The second era came about in 2001 when Action-oriented frameworks became popular. In 2004, JSF and friends came to play. JSF is a component-based framework with known limitations (complex, slow, uses POST for almost everything, Ajax is difficult). These limitations have resulted in a number of third-party patches being developed to solve these issues.

JSF was designed to use the request to create a component tree that maintains state. Unfortunately, the state is not something the developer has control over. It's not the state of the application, it's the state of the components. The client's knowledge of the state is mentioned with a cookie and the server keeps the state in the session. The problem with JSF is you don't have a choice of state in your application - you can't write stateless applications like you can with servlets.

JSF uses overloaded URIs for its resources. When you have one URI with multiple representations, there's no way to tell how a representation was chosen. JSF's compromise is to allow client-side state saving. However, they do this by putting hidden field in the form and requiring POST for navigation.

JSF vs. REST
Basically, these two are at opposite extremes. JSF is focused heavily on a UI component model. The people that developed it misinterpreted how the web works and made some fundamental questionable choices. You can patch it, but you can not fix it.

Web 2.0 Frameworks
GWT is a cross-compilation based framework. You write Java to generate JavaScript (b/c everyone hates writing JavaScript). It mashes client and server code into a single source. These layers communicate using GWT-RPC. Typical RPC concerns do not apply since code generation handles coupling and the client is downloaded from the same application. GWT-PRC does POSTs to the server and uses HTTP like a transport layer. To be fair, GWT does allow you to use a RequestBuilder to use the web like it should be used. This class allows more control over HTTP requests, it supports GET and POST and it allows so-called RESTful layers (GWT-REST and GET-Restlet). GWT is focused heavily on ease-of-use, which is good. It's modeled after RPC and breaks the uniform interface and focuses on backend caching. Unlike JSF, GWT is fixable, but the community tends to use RPC instead of RequestBuilder.

SOFEA has a central promise of SOA. Business logic is a reusable service that changes less often. The presentation application calls those services and changes more often. The nice thing about this type of architecture is it allows a separation of concerns and loose coupling. However, it doesn't embrace REST like it should. Appcelerator is an implementation of SOFEA that has a Ruby on Rails-like usability. However, it uses a SOAP/HTTP style with messaging and POSTs to a single URI. Appcelerator is interesting, but it introduces a different style of coupling. It breaks URI opacity and client deals with POX instead of links.

Conclusion
Don't fight the architecture of the web. Innovate and enhance instead of breaking. If nothing else, break judiciously. As developers, we should demand more from our frameworks and make sure they use the web and HTTP like it should be used.

Ajaxified Body

I've often wondered if it was possible to use Ajax to reload the main content of a web application without reloading the header, menu and footer. SiteMesh allows you to move these common elements to a decorator that gets wrapped around each page. Below is a diagram of how SiteMesh works.

You can read the Introduction to SiteMesh article if you're interested in learning more about SiteMesh's basic features. By default, SiteMesh decorates text/html responses and ignores any other content type (e.g. image/gif). It also contains an <excludes> configuration element that allows you to turn off decoration when a URL matches a certain pattern. For example, the following allows you to disable decoration when "ajax=true" is passed in the URL.

<excludes>
    <pattern>**ajax=true</pattern>
</excludes>

To optimize the loading of an application so the common elements aren't loaded for each page, it should be possible to create an Ajaxified Body where the primary content area (PCA) of the site is loaded via Ajax rather than refreshing the whole page. The header, footer and navigation bar often stays the same from page-to-page, so it doesn't seem to make a whole lot of sense to load them each time the page loads. The image below shows the PCA (of an AppFuse Light application) as a grey square with a red border.

Implementing an Ajaxified Body consists of the following steps:

1. Adding SiteMesh and moving common elements to a decorator.
2. Remove common elements from each individual page (if you're using includes).
3. Configure SiteMesh so decoration is disabled when the requested URL contains "ajax=true".
4. Write JavaScript that modifies all <a href=""> links (and buttons with onclick='location.href') in the PCA to have an onclick handler.
5. The onclick handler should call a JavaScript function that loads the link's URL + ajax=true using XMLHttpRequest (XHR).
6. Add XHR success handling to replace the PCA with the loaded content.
7. Add XHR error handling to go to the URL normally when response.status != 200.
8. Inspect the response HTML for <title> element and replace document.title if exists.
9. Inspect the response HTML for <head> element and append to current if exists.
10. Inspect the response HTML for <script> and <link> elements (JavaScript and CSS) and evaluate them if they exist.

As a proof of concept, I created a prototype using AppFuse Light (Prototype/Scriptaculous for Ajax). You can see a demo at the following URL. You can also download a patch or the source for this project.

http://demo.raibledesigns.com/ajaxifiedbody

Below are a number of things I discovered while writing this prototype:

• The hardest part of implementing this seems to be coding the exceptions. It's possible you'll have some links with existing onclick handlers and you may have to disable "ajaxifying links" for those links.
• A progress indicator is important or the page might load so fast that the user doesn't visually detect it changed. This can lead to a worse user experience because they don't see the flash of the blank page they're used to when a page load occurs.
• While forms can be submitted via Ajax, there's no harm in leaving existing form behavior in place where the full site is reloaded after submitting a form.
• If a particular page needs to change the common elements (header, menu, footer), it should be possible to do that with JavaScript after the PCA content loads.
• If the success/error indicator is outside the PCA, it may need to be populated and displayed/hidden with JavaScript after the PCA loads.

I'm sure my implementation can be improved, but I'm also curious to see what you think of this idea. I know it's not revolutionary, but it's something I'm considering adding by default to AppFuse and AppFuse Light. Do any Ajax frameworks do something like this out-of-the-box?

Update: Thanks to everyone for the great feedback - keep it coming. I agree that adding history support is a must. I'll try to do that in the next day or two. This post has also been featured on Javalobby and Ajaxian.

Update 2: Added history support.

Raible Road Trip #12 Vacation Report

I'm happy to report that the kids and I made it home safely last night after a heckuva Road Trip. We were gone for a total of 8 days and we drove for 4 of those. Here's some stats from Snow White:

• 43 Hours
• 2248.3 Miles
• 150.9 Gallons of Gas
• 14.9 AVG MPG
• 52 AVG MPH

Of course, these stats include day-to-day driving while in Montana.

Driving to Montana
We left early on Saturday morning, waking up at 4:30 and were on the road around 5. This was brutal considering I went to bed 4 hours earlier the night before. My Dad and I took turns driving and tried to listen to an audiobook (Spook Country) along the way. However, whenever one person wasn't driving, they fell asleep so the whole idea somewhat failed.

We arrived in Yellowstone around 5PM and checked into our campground. The ranger told me there were 5 grizzlies in the campsite and that we should pack up our food and put small pets in cars for the night. Of course, I couldn't wait to tell this to Abbie, who is always scared that bears will attack us when camping. I finally calmed her fears by telling her I wrestled a bear once when I was a kid and won pretty easily. After setting up camp, Abbie took a swim in the "freezing cold" Yellowstone Lake and we ate dinner and climbed into our tent shortly after.

On Sunday, we woke up and made it out of our campsite by 7:30. We drove to Old Faithful, enjoyed an eruption and proceeded to have a nice breakfast at The Old Faithful Inn. After checking out some more geysers, we made it out of the park around 11. We stopped briefly in Bozeman to see an old friend, hit Clearwater Junction and Lucky's (for huckleberry milkshakes) around 6, and made it to The Cabin just before dark.

At The Cabin
The few days we spent at The Cabin and in the Swan Valley were a lot of fun. I got to drive the family Excavator with both kids on my lap. Jack was operating the bucket while Abbie was operating the swivel of the cab. I took my mountain bike, so I got to ride some old trails I used to ride all the time as a kid. Not only were there beautiful views on the trails, but I also got to experience quite a wreck that sent me head-first into some rocks and bushes. It was the first time I've wrecked in a while where I thought to myself "that was fast and hard enough to break some bones". Luckily, all I received were a number of scratches and bruises.

On Thursday, we took my sister to the train station in Whitefish. We left early enough for my dad and I to get in some golfing at Meadow Lake Golf Course and the kids to hit Big Sky Waterslides. Apparently, they didn't have a height requirement because they were able to go on all the slides.

On the 4th, we finished Abbie's Princess float in the early morning, took a quick sauna and headed down to the Swan Valley Parade. We'd been talking about the parade and Abbie's float for several weeks, so I was kinda nervous that I might mess it up. Not only did everything go great, but there were several hundred folks that clapped when we drove by and commented on what a beautiful princess Abbie was.

After the parade and drinking some good ol' Busch Light from the "Beer Garden", we headed to Holland Lake for a swim and canoeing. We drove back home in time to play a game of horseshoes and get the fireworks setup for the evening. We had a good fireworks show (with yours truly in charge) and enjoyed lots of laughs with old friends.

The Drive Home
As usual, the drive home was the longest portion of the trip. The first day, we left the cabin around 11 and pulled into Billings around 5. The kids (and dog) did pretty good as we only needed to stop once. The shorter first day turned out to be a good idea since the kids like hotels so much. Yesterday, we left Billings around 9:30 and cruised along mightily until we hit the Colorado border. There, traffic came to a standstill and we suffered through stop-and-go the rest of the way home. After 9 hours, we pulled into my driveway, exhausted.

I don't believe we'll be driving next year. The Road Trip to The Cabin is something that should only be done every couple years. Snow White continues to be an awesome car. We towed a sawmill (on a trailer) to The Cabin and didn't even feel the load. With DVD screens in the back of both front-seats, it was an excellent road-trip vehicle for the kids. Also, it's smooth suspension made for an enjoyable experience for everyone (we drove Julie's Honda Odyssey one year and Abbie kept getting sick from the DVD screen bouncing). The more I drive it (esp. skiing, camping and on road trips), the happier I am with the purchase.

For more photos from the trip, please see Raible Road Trip #12 on Flickr.

Good times with kids in Colorado

A friend recently sent me an e-mail asking the following question:

May be coming to Colorado in early August for a good break. Can you share ideas on having good times with kids?

His kids are 3 and 5 years old, the same as Abbie and Jack. Here's my response:

[Read More]

What's a good RIA to develop in 20 hours?

In less than two months, I'm making my annual trek to Portland, Oregon to speak at OSCON. To prepare for my talk, I'd like to develop the same application with two different combinations: Flex + Rails and GWT + Grails.

As luck would have it, I'm having a hard time coming up with a good application to write. I'd like to time-box it so I only spend 10 hours on the backend (for each) and 10 hours on the front-end, for a total of 40 hours for both applications.

Can you think of any good applications that would warrant a rich front-end and wouldn't take too long to create? I'd like to put both applications in production and generate enough traffic to be faced with scalability issues.

Over the next several weeks, I hope to start creating the applications and blog about what I've learned along the way. At some point, I hope to post an outline and a rough draft. With your help, I believe this can be an excellent presentation. If the presentation and applications are as good as I hope they'll be, it's likely I'll open source them for everyone to use.

Thanks in advance for any advice.

Update: Thanks for all the great feedback. I've posted a survey to narrow the choices.

2008: The Year of Beer

In a few weeks, I'm heading to the American Craft Beer Festival in Boston. In July, I'll be attending the Oregon Brewers Festival in Portland (followed by a night at McMenamins Kennedy School). A friend and I just booked flights for Oktoberfest in Munich. A couple weeks after Oktoberfest, I'll be attending the Great American Beer Festival in Denver.

Homer would be proud.

The Web Framework Smackdown Questions

I'm doing my Web Frameworks Smackdown this morning at TheServerSide Conference. A few weeks ago, I asked What Would You Ask the Web Framework Experts? on Javalobby and LinkedIn. Here's a summary of those questions:

• What is the overall performance of your framework as it compares to others?
• How does your web framework position themselves in relation to Web Beans?
• How easy is it to create a re-useable component in your framework? Is it as easy as sub-classing an existing component?
• What is the key differentiating characteristic of your framework that makes it better than the rest?
• What do you think about the various scopes introduced by Seam, e.g. conversation vs request or session? If you support these additional scopes, do you also provide some sort of concurrency control?
• Why can't we, the Java Community, come together and adopt the best application framework and settle the web development subject?
• What are you doing to help with developer productivity?
• 2008 is a huge year for the mobile web. How do you help developers build great mobile web applications?
• If you couldn't use your framework, what would you use and why?
• How do you enable rich Ajax applications?
• Can a developer make a change to source, and hit RELOAD in the browser to see the change? If not, why not?
• What do you think about the whole Flex revolution, and do you think you are competitors to this technology?
• How easy is it to create a module and plug it into a bigger application, complete with configuration, code, and view?

Of course, there's many more questions on the aforementioned pages, these are just some that I hope to ask during the panel. Sitting on the panel: Don Brown (Struts 2), Keith Donald (Spring MVC), Ed Burns (JSF), David Geary (GWT), Geert Bevin (RIFE/OpenLaszlo) and Justin Gehtland (Rails). I tried to get Flex and Grails folks, but they'd either left the conference already or are speaking at the same time.

Update: InfoWorld has some modest coverage of this event in Web frameworks debated at TheServerSide Java Symposium.

Audi A6 Quattro

A few weeks ago, a friend recommended I join Hertz #1 Club Gold. His reason was simple - you can walk out of the airport, read your name on a board, jump in the car and go. No talking to anyone, just get your rental car and you're on your way. It sounded like a good idea, so I signed up.

Last week, I went online to reserve a car for this week's trip to Mountain View. I chose the cheapest car, and was subsequently prompted to choose an Audi A6 from their "Prestige Collection" for less money. I went for it.

This morning I hopped in the car at SFO and was immediately impressed. It's probably one of the nicest cars I've ever driven. It's not as fast as the Cayenne Turbo I drove for a week in Florida a couple years ago, but it's certainly a slick car. You can even talk to it to get it to change radio stations and such.

It's too bad I only have the car for one more day. I'm heading to TSSJS tomorrow night in Vegas. I wish there was a way to cancel my flight and drive instead.

Jack's got a bead stuck in his nose!

This morning around 9:45 AM, I got a call from Jack's Teacher. She said, "Jack's got a bead stuck in his nose!" I heard a screaming kid in the background, so the first thing I asked was - "Is that Jack?" She said yes, and I was out the door a few seconds later.

Several minutes later, I arrived at his school and picked him up to take him to the Emergency Room. His teacher handed me the bead you see in the picture on the right. This was a replica of the one he jammed into his nose. I figured the ER was the best place to go considering all I saw was blood when I looked in his nostril.

10 minutes later we were in the ER and within a half hour we were talking to a nurse. Julie arrived just as the doctor walked up to talk to Jack. The first thing he suggested was that Julie hold one nostril and give him a CPR-type breath/blow into his mouth. She tipped his head back, plugged the free nostril and "pop!" - it came right out!

I wish I'd known that trick early this morning. How cool would it have been to walk into Jack's class, grab him by the head and blow that sucker out like I knew what I was doing? The good news is now you have this knowledge and you can be the hero in your kid's class someday.

Last Night's Selenium Users Meetup at Google

Last night, I attended the inaugural Selenium User Group meetup at Google's Campus in Mountain View. It was an excellent event, with many of the core committers on hand to present and answer questions. Each presenter had about 5 minutes to speak and we learned many things about the Selenium Project itself, what's coming in the future and how Google has standardized on Selenium as their integration testing tool of choice.

Patrick Lightbody started the meeting by going over a number of project statistics with pretty graphs and such. There were simply too many numbers to write down, so hopefully his slides will be published soon. I was pleased to see that Google did videotape the entire event, so it should be available online soon. I'll update this post when it it. Below are my notes from the event.

Jason Huggins, Test Engineer at Google
Selenium is a test automation framework. Some folks abuse it as a macro tool. There's two reasons Selenium became so popular: it was able to test Ajax before any other testing tool and it allows end-to-end workflow testing. Selenium works on any platform, with any browser and allows many, many languages. It's possible that other frameworks that are more focused are better. There are 4 Selenium products:

• Selenium Core (TestRunner)
• Selenium IDE (for Firefox)
• Selenium Remote Control
• Selenium Grid

Paul Hammant, Open Source Manager at Thoughtworks
Selenium 1.0 - beta this week. RC, Core, Grid and IDE together. 1.0 will be shipped in a few weeks. Compared to today, lots of bugs killed, documentation improved and a greatly improved Selenium IDE.

Selenium 1.1 will be shipped in a couple months. Selenium IDE will be enhanced to obey RC instruction ... becoming the best mode of operation when it ships.

Some experimental iPhone-Safari capability (dependent on SDK for iPhone).

Selenium 2.0 will hopefully be released this year. Roll in of WebDriver functionality and code. New model-based API. IE plugin and Safari plugin (not really a plugin, most likely uses AppleScript).

Philippe Hanrigou (http://ph7spot.com)
Testing is good! Selenium is awesome! However, end-to-end web testing is slow. It is and always will be slow. How do you solve this problem? You solve it the same way we've solved slow traffic in the past - by building more lanes. Rather than one browser testing everything, Selenium Grid allows multiple lanes and 20 browsers. Features include:

• Faster Builds, Faster Feedback!
• Easy Install and Everyday Use
• No Need to Change Your Tests
• Leverage Your Existing Computing Infrastructure

One of the best parts about Selenium Grid is you can download and have it running in 10 minutes or less. Selenium Grid comes out-of-the-box with Amazon EC2 support.

Jennifer Bevan, Lead from Selenium Farm Project at Google
As of Friday, Google has over 50 teams running over 51K tests per day on internal Selenium Farm. 96% of these tests are handled by Selenium RC and the Farm machines correctly. The other 4% are partly due to RC bugs, partly to test errors, but isolating the cause can be difficult. Selenium has been adopted as the primary technology for functional testing of web applications within Google. That's the good news.

The bad news is Google is pushing the limits of Selenium. Using Selenium (RC + Core) at this scale exposed certain issues not originally anticipated as high-impact. IE/XPath slowness has a high impaction given that can only run one test at a time. Tests can cause many conditions from which RC cannot easily or automatically recover (for example, tests that don't call stop() in every exit path). Unexpected browser dialogs, popups, etc. eventually cause timeout exceptions.

Googlers expect that RC will work for the most part, but they want it to be more reliable, with better performance. So they have:

1. Created utility methods to improve performance when examinging large tables, overlaying domain-specific languages, etc.
2. Deployed retry policies based on failure reason.

Looking to the future - Google has not yet reached their expected usage of Selenium RC. Some projects cannot use the Farm until RC supports session-level configuration (not server-level). Many just want RC to be more reliable. So Google will:

1. Continue to contribute to RC, Core and to user-created helper libraries.
2. Keep doing so until all failed tests are not Selenium's fault.

Dave Astels, Google (Driving Selenium with RSpec)
Using RSpec, you can create a very easy-to-read Story with Scenarios that can be read (and likely written) by practically anyone. Dave then uses a small script to load up the stories and run them in Selenium. When he runs the script, the scenario is spit out and test pass/fail information. Learn more at rspec.info.

Alex Chaffee, Mad Scientist at Pivotal Labs (The Selenium/Ruby Project that Must Not Be Named)
What is Polonium? It's also known as Selenium RC Fu or Selenium On Rails 2 or Funkytown. It has simple extensions to Selenium RC:

• wait_for
• element assertions
• launching/managing servers locally

Blackbox testing you're sitting out of the box and send in stuff. In whitebox testing, you get to open up the box and look at stuff. With Selenium, you can do Graybox testing, where you are doing blackbox testing (against the UI) and querying your database (or other resources) at the same time.

Dan Faulich, Sr. QA Engineer at Redfin Corporation (How Not to Run a Successful Open Source Project)
The first thing you don't want to do is support everything. The second thing you don't want to do is write your project in JavaScript. While it's great that almost anyone can hack on it, running in a sandbox sucks. Don't use a language that's bound by other people's security restrictions. Don't roll your own multi-language remoting. It's written using XML + XSLT and its such a pain in the ass to maintain that Dan is the only one that fixes bugs.

Shinya Kasatani, Developer of Selenium IDE
Selenium IDE is a Firefox extension that can record and play back tests in your browser. It can translate the recorded tests to many languages.

Selenium IDE 1.0 adds support for TestSuites. Another new feature is better recording features - it detects when the DOM is modified. Shinya has a demo where he uses the new IDE to test the Dojo Dijit Theme Test Page. Apparently, this doesn't work in the current version.

The Goal of Selenium IDE is to get more people interested in test automation of web applications and to help their projects to be successful.

Haw-bin Chai, Developer at CommerceHub
XPath is a powerful selection took, but it'd be great if we could use something like "article 5" instead of the cryptic //table[5]/tbody/... syntax.

Why UI-Element? Traditional locations can be ugly and they don't convey purpose. UI-Element is a Selenium Core extension and has Selenium IDE integration. It's written in 100% JavaScript. Examples:

ui=frontPage::topStoriesCountry()
ui=listingPages::article(index=5)
ui=listingPages::articleSource(articleIndex=1)

The UI Map is in JavaScript shorthand. It contains logic to map ui locators to traditional locations. Each UI element implements getLocator(), then you add a page set and add an element - all in JSON Format.

Simon Stewart on Web Driver
What on earth is Web Driver? One of the main problems with Selenium is it's written in JavaScript and it's too easy to hack. WebDriver is an idealized browser which allows you to do browser things like call get(URL), findElement(By.xpath()) and getTitle(). WebDriver is similar to Selenium RC, but it's not written using JavaScript. It's written in the native languages for each browser, which allows you to break out of the JavaScript Jail. The IE Driver is written in COM (C++). The Firefox Driver is written as a Firefox extension. Safari uses Apple events.

Soon there won't be a WebDriver project ... because it will be part of Selenium!

WebDriver likely won't be part of the core until Selenium 2.0 later this year. One of the nice things about WebDriver is you can implement different browsers. Out-of-the-box, it will support all the major browsers, as well as HtmlUnit with JavaScript turned off.

I had a great time learning more about Selenium and how most of its problems will be solved in the near future. The beers afterwards weren't so bad either.

Update: Videos of this event have been posted.