How I Calculated Ratings for My JVM Web Frameworks Comparison
When I re-wrote my Comparing JVM Web Frameworks presentation from scratch, I decided to add a matrix that allows you to rate a framework based on 20 different criteria. The reason I did this was because I'd used this method when choosing an Ajax framework for Evite last year. The matrix seemed to work well for selecting the top 5 frameworks, but it also inspired a lot of discussion in the community that my ratings were wrong.
I expected this, as I certainly don't know every framework as well as I'd like. The mistake I made was asking for the community to provide feedback on my ratings without describing how I arrived at them. From Peter Thomas's blog:
What you are doing is adjusting ratings based on who in the community shouts the loudest. I can't help saying that this approach comes across as highly arrogant and condescending, you seem to expect framework developers and proponents to rush over and fawn over you to get better ratings, like waiters in a restaurant trying to impress a food-critic for Michelin stars.
I apologize for giving this impression. It certainly wasn't my intent. By having simple numbers (1.0 == framework does well, 0.5 == framework is OK and 0 == framework not good at criteria) with no rationalization, I can see how the matrix can be interpreted as useless (or to put it bluntly, as something you should wipe your ass with). I don't blame folks for getting angry.
For my Rich Web Experience presentation, I documented why I gave each framework the rating I did. Hopefully this will allow folks to critique my ratings more constructively and I can make the numbers more accurate. You can view this document below or on Google Docs.
In the end, what I was hoping to do with this matrix was to simply highlight a technique for choosing a web framework. Furthermore, I think adding a "weight" to each criteria is important because things like books often aren't as important as REST support. To show how this might be done, I added a second sheet to the matrix and made up some weighting numbers. I'd expect anyone that wants to use this to downloaded the matrix, verify the ratings are accurate for your beliefs and weight the criteria accordingly.
Of course, as I and many others have said, the best way to choose a web framework is to try them yourself. I emphasized this at the end of my presentation with the following two slides.
You probably have direct access to the Linked-In database, so your queries may be more accurate, but a quick search on the site gets 1,593 results for 'java wicket', and 1,443 results for 'java tapestry' (I included 'java' to avoid cricket players and carpet weavers) so, I don't see a reason for the difference between the two in 'Developer Availability'.
Both wicket and tapestry have very similar counts on dice.com ('java wicket'=45/'java tapestry'=49), and on indeed.com, wicket surpasses tapestry on absolute numbers, and beats it to the ground on relative growth (again, 'java wicket' vs 'java tapestry'). Still, wicket's got .5 while tapestry' got 1.0 on 'Job Trends'.
That's only two numbers that won't change much the results, but that reveal some lack of consistency of your grades and your (supposedly objective) explanations.
Posted by Tetsuo on December 06, 2010 at 09:49 PM MST #
Matt, I'm not trying to troll here, but since we've (or you and Peter Thomas rather) have had a few back and forths, here are my objections to your approach.
* Developer Productivity
While developer productivity is one of the most important things to look at when selecting a framework, you reduce it to a simplistic reloading feature. Much more useful things to look at would be whether the framework lets you divide tasks amongst a team well, whether the code you write for it is easily maintainable (is it easy to read, can be it be easily explored and refactored, etc), does it encourage reuse on different levels of granularity and does it let you avoid code duplication well. Questions like these are far more interesting than just whether you need to restart often.
So, it's the question I disagree with. Then the answer to this particular question. Wicket (my expertise as you may remember) requires less restarts than some others it is scored equally to just because templates are automatically reloaded (in development mode) and non-structural changes are reloaded in the JVM. So even a simplistic question like this isn't answered well.
* Developer Perception
If you *really* want to include something like this, make it statistically sound, and also include how it developed over the years, what versions etc.
* Learning Curve
I've said this before, it depends where the developer picking said framework up comes from. In my experience - not imagined or theorized, but something I have seen several times - if people haven't been using web mvc frameworks, but do have Java/ OO experience and maybe done some desktop UI coding, it is actually easier to pick up frameworks like Wicket, Vaadin and GWT.
It's funny that you note that SpringMVC and Struts 2 etc are easier to learn *for you*, but you are not writing this comparison just for you, are you? Exactly my objection to your comparisons in general: they are subjective made to look objective.
* Project Health
Fair enough, but as soon as you're looking at just volume you're discounting smaller frameworks that may be incredibly healthy but just comparatively small. And then there is that good old "more questions isn't always a good thing" thing we've discussed several times in the past. And do you measure how many people give answers? How good those answers are? The involvement of committers and project leads?
* Developer Availability / Job Trends
Oh please. I am involved in hiring people all the time. Whether they have experience with a certain framework is rarely an issue. People not understanding the underpinnings of technologies (like what problems does DI solve... surprisingly the majority of people don't seem to have a clue) or people being over-specialized in particular ones are often our reasons for rejection.
* Templating
Yes, very good thing to look at. But score it? GWT 2.x is the default now and that supports decent templating. For Vaadin, not supporting templating is a design choice; how can you give a bad rating for something they particularly chose to differentiate their framework? From their perspective (playing the devils' advocate), template based frameworks are lousy because it's all a bunch of string mongering.
* Components
Useful question again. But why does GWT get low marks? Reuse is easy and reasonably elegant. Much better in my opinion than JSF, which gets a higher mark.
(from here on, due to time constraints just going to cherry pick a few of the remaining headers)
* Scalability
Useful point, but incomplete analyses. Yes, it a drawback to scalability if a framework depends on servlet sessions. However, in practice it takes a lot of discipline for you average Java dev team to stay clear of this, especially using frameworks like Struts, and frameworks like Wicket or Vaadin which rely on session state, but also manage it tightly, often results in smaller actual session usage. Again, something I've seen in practice, and commonly heard is other's people experience (Peter Thomas also has an article that includes this note).
Also, a server side state model often results in less chatter and better locality (in particular when you have interrelated widgets that have an overlap in the data they need, it is cheaper to do this in one update rather than many requests like you'd have with a more Ajax-centric approach).
* Books Published
Wicket has 4 (or more if you take language specific ones into account) books, and that gets it a 0.5? Regardless, I think this should be part of your overall documentation item, and you should care less about how many books there are, and more about how good they are and if there is at least one book that most developers are really helped with.
* REST Support (client and server)
I don't agree with your assumption that more is better here. I use Jersey/ JAXRS together with Wicket and GWT for the projects I work on, and that works great. I rather mix a few best-of-breed frameworks, than depend on frameworks to do everything for me.
* Mobile / iPhone Support
Same as previous.
* Degree of Risk
Too simplistic to just look at the number of developers and how new the project is. Projects with too many developers are equally risky when it comes to potential splits and stagnation, political issues, etc. I would never include this item unless there is something particularly risky about a project (for instance, it exists less than a year and hardly has any activity on the list).
And then there is that you don't include some very important things. Security for instance. A side benefit for Wicket's server side state model, and actually one of the main design decisions for Vaadin is that storing all state on the server gives you a much more secure programming model right out of the box. Not mentioned anywhere. The kinds of functionality a framework comes out of the box with. How easy it is to customize (e.g. custom error and URL handling, up to pluggable template parsing etc).
Anyway, after this long list of objections, I would like to give kuddos on the "choose your own" slide. Maybe if you'd start with that, repeat it in the middle and then note that again at the end, you wouldn't get heated replies like this as often :-)
Posted by Eelco Hillenius on December 06, 2010 at 09:56 PM MST #
@Tetsuo - thanks for taking the time to lookup these numbers. I no longer work at LinkedIn, so I imagine you have the same access as I do.
I recorded the numbers I found when creating the presentation. It appears you are correct - searching for "Tapestry" vs. "Tapestry Java" yields much different results. I agree with you that Tapestry and Wicket should have the same rating for Developer Availability. They should also have the same 0.5 ranking for Job Trends since they have less jobs than frameworks like Rails and GWT.
Posted by Matt Raible on December 06, 2010 at 10:10 PM MST #
Matt, those last slides are key. I've always viewed your (interesting and valuable) presentations as subjective anyway. I've disagreed with a few points, but the main take away from your presentation (for me, at least) is that if you are going to evaluate and compare web frameworks you should try to compare frameworks methodically. If someone is attending your talk with the expectation that you have adhered to rigorous, scientific standards then they are missing the point entirely.
@Eelco, please post a response of that size on your own blog. That's not a comment, that's a reaction.
Posted by Tim O'Brien on December 06, 2010 at 11:17 PM MST #
Posted by Eelco Hillenius on December 07, 2010 at 01:00 AM MST #
Matt, appreciate the response. Apologies for the admittedly unkind response, but yeah, I was upset. It is easy for people like Tim O' Brien to pontificate about how conference attendees should behave in theory, but he doesn't have to deal with pointy-haired bosses forcing you to use a framework in a new project because Matt Raible said good things about it at this one esteemed conference ;)
Oh, and now, back to work.
Posted by Peter Thomas on December 07, 2010 at 03:21 AM MST #
Posted by Craig Tataryn on December 07, 2010 at 05:48 AM MST #
Matt, kudos on the great work of the web framework comparison. It's been helpful for our team here at VolunteerMatch both as a discussion tool and to take it, adjust ratings where we feel differently, and see where we come out.
If the complainers would put their energy into making their own "better" version instead of focusing their energy on reacting to your version, the developer community would be a better place.
When I hear the complaining it reminds of that SNL skit at the post-office and a dis-satisfied customer where the postal worker says "If you can find someone else to mail your letter for 25 cents then I suggest you do so".
Posted by Todd Huss on December 09, 2010 at 08:02 PM MST #
Posted by Ondrej Medek on February 04, 2011 at 01:15 PM MST #
Posted by Pedro Barroso on July 14, 2013 at 04:16 PM MDT #