[DJUG] Building an Open Source ESB and Ruby on Rails
Managing Chaos
Building an Open Source Enterprise Service Bus from Scratch
Bruce Tate got rained out in Texas, so Bruce Snyder and Jeff Genender are talking about building an ESB with open source software. Bruce and Jeff created this product in the last project they were on. I was lucky enough to work with them on it the first half of this year - but only after the ESB was created. The final product was very stable and the client loved it.
Bruce is a Geronimo committer and one of the founders of the Castor project. Jeff is also a Geronimo guy and is currently working on a JBoss Live book for SourceBeat. The problem that the ESB was trying to solve was fixing a horrendous data flow. A lot of the data flow was occurring between people's PCs, shared drives, FTP servers, HTTP Servers, web servers - and rarely were things automated. Bruce recalls one of his first days when he heard the trading guys yelling at each other to "close the spreadsheet".
The solution to the problem was building an Enterprise Service Bus (ESB) that performed the following:
- Centralized Management of Activities
- Powerful Scheduler
- Guaranteed Event and Activity Execution
- Durable Transactions
- Pluggable ESB Components for Activities
- Staging Database for Single Common Data Location
- Logging and Notification of Activities
- Proactive Response to Failed Activities
- J2EE Architecture - Provides for True 24/7 Uptime
The pluggable components were called transformers and were standalone JARs that lived on their own, but could be managed by the ESB. Notifications were key so the traders would be notified when something went wrong.
Architecture
Scheduler (Quartz) » Workflow (jBPM) » Persisted Guaranteed Messaging (JMS). JMS talked to Activities (a.k.a. transformers).
The Quartz Java Scheduler is an open source project from OpenSymphony. It's a persisted scheduling engine, so it'll live through app server restarts. It also has millisecond granularity.
For workflow, the Java Business Process Manager (jBPM) was used. It doesn't use BPEL, and was used to track multiple activities and make decisions based on an activity's completion or failure status. Other functionality included tracking the activity state (running, cancelled, completed) and sending/managing notifications. Workflow was very important because the previous system had no way of detecting where things failed in a process. With the new system, downstream dependencies were handled, the escalation path was based on success or failure - and automatic retry occurred on failures if the failure reason was a known and configured expectation.
For messaging, JMS was used - implemented with EJB and MDB. This provided guaranteed and persisted messages. Events were automatically recovered if the server failed.
Activities/Transformers were pluggable components (wrapped with EJBs for transactionality). The nice thing about using EJBs was it was easy to create JARs for each transformer, drop them into JBoss and they'd immediately become available.
The system was all managed with a management console, that was a webapp implemented in Struts, Spring and Hibernate. Most of the Spring and Hibernate classes and configuration was generated with Middlegen and XDoclet. The management console allowed you to kick off activities, monitor their progress, as well as manage users with Active Directory and single sign-on with NTLM and jCIFS.
Solution Facts
Since December 2004, over 200,000 jobs have been run through this system. Of those, 500K activities have been run, with < 10,000 failed activities (2% failure rate). Nearly all failures were due to external issues, such as unavailability of remote systems and databases, or source files not available.
Tools and APIs Used
When Jeff showed up, it was a Microsoft shop using ASP, Visual Studio and some PowerBuilder. They brought Jeff in to help them use and adopt open source. The first thing they did was install and begin to use CVS (previously source control was done on shared drives). They also used Maven to build everything and produce a project site - which the managers and C-types loved. One thing they mentioned is they often got questions from higher-ups like "How much does it cost?" They did have problems with Maven, but it was mainly due to the poor documentation. They found a lot of Maven solutions by cracking open plugins and looking at their Jelly files.
Development Lessons Learned
Configuring EJBs and MDBs as singletons helped solve some problems (a JBoss setting allows you to configure this). As for running Spring in a heavily-managed environment, they found that setting singleton="false" solved a lot of problems. The next problem they had was mixing Hibernate and JTA Transactions. No details, just that they had an interesting time and it took them a few days to get it working. The last problem they encountered was using Hibernate and/or Spring JDBC to manage hundreds of thousands of records. Since these O/R tools create objects for each record, OOM errors occurred with large resultsets.
Business Lessons Learned
All notification messages came from the ESB, leading many to believe the problem was the ESB - rather than the data sources it was talking to. By using open source, they saved the company $500K in licenses fees. The interesting part was the company had a 3rd-generation agreement with IBM, and owned $12 million worth of licenses for WebSphere and WSAD. The reason this group used open source was because there weren't enough licenses.
Bruce and Jeff's presentation was good, but they looked like a couple of goofballs
standing up there in their t-shirts and shorts (standard Virtuas gear). I
guess that's what happens when you get a 2-hour notice. The worst part about
the presentation was the fact that the A/C doesn't work and it's about
85° in here. I told Geary he'd better keep in short or I'm outta here.
Ruby on Rails
David Geary
David got into Rails by reading an article called "Rolling with Rails". In the article, the author claimed that you can develop webapps in Rails at least 10 times faster than in Java. David responded to the article on his blog with an entry called the Ruby on Rails Koolaid. He experienced quite a butt-whooping from various folks, including Rails' founder - and realized afterwards that the claims might be valid. After working with Rails, David believes that Rails is probably 5-10 times faster.
Ruby
Potent mix of SmallTalk, Perl and Python. Everything is an object. No static type checking. Duck Typing (talks like a duck, walks like a duck, it probably is a duck). Testing usually solves the lack of static types. Blocks - like anonymous inner classes, but retain state. Mixins - a cheap way of doing multiple inheritance. Dynamic classes - can modify at runtime (add methods, renaming methods, etc.). Rails takes great advantage of the dynamic attributes as part of its framework.
David is now showing a ContactsController that has 5 methods for
CRUDing a Contact object. 4 of the 5 methods are one-liners. It kinda reminds
me of my Hibernate DAOs after integrating Spring.
Rails
Ruby-based MVC framework. Convention over configuration. Scaffolding - builds pieces of your application for you. ActiveRecord does O/R Mapping. Has a built-in testing framework. Near-zero configuration (no XML). Zero-second deploy time (development environment).
David's first demo is being done by audience member Kirk. David asked for
a volunteer from the audience with MySQL experience, and the ability to type
in a few commands. David said his 6-year old daughter was able to do this
demo last night successfully - so Kirk's gonna look pretty bad if he can't
pull it off.
Using scaffolding, Rails generates 1 controller, a test for it, a helper class, a css file and 5 rhtml templates. Kirk pulled off the demo, even though he had a bit of trouble with the Mac environment. This is a lot like AppFuse's AppGen in a sense - except AppGen has to parse a bunch of XML files and reconfigure them.
David keeps hammering that the most productive feature of Rails is that there is zero deploy time. Save. Refresh. It looks very similar to developing a static HTML site.
Rails has ActiveRecord, ActionPack (MVC), ActionMailer, ActionWebService, Ajax Support, Transactions and Security. Currently at version 0.13.1 - the last version before 1.0.
David is delivering an excellent presentation, but it's too damn hot without A/C - I'm outta here.