Raible Designs

Notes from DJUG: Rhythm and Open Source Strategies

Last night I had the pleasure of listening to two great speakers at Denver's JUG. Not only did Brian and Lajos know their stuff, but they were very comfortable on stage and interesting to listen to. Below are my notes from this event. When I did a print preview of this in Dreamweaver, it came out to be 5 pages - so just to warn you, this is a pretty long post.

Brian Boelsterli - Nudging your organization towards agility: Ingredients for an agile execution environment

Even if you have a dream team of developers, it doesn't mean the project will succeed. Process Matters. A lot of what Brian's Rhythm process was developed while working at Sprint - where they completed 169 iterations on their way to production. There are problems with agile methodologies: team has no rhythm (restart, pause, delay all the time), project plans are big and heavy and just about worthless, methodology teams get in the way, specifications are close to worthless, design review meetings are horrible. In order to combat these problems (regardless of your methodology), Brian has developed a Rhythm Process that helps agile methods succeed. This system contains a number of ingredients to solve problems with agile methodologies.

The Heartbeat is the logical representation to the physical thing called an "iteration". Starts Thursday morning, ends Wednesday night - based on psychology b/c it fosters sustainability. By giving 2 days to start a new iteration, they've found higher productivity because no one wants to work the weekend.

Rhythm advocates weekly iterations, regardless of deliverables. The primary reason for this is to maintain the heartbeat. Rather than doing XP-type iterations where you do everything in an iteration, you do Stacked Iterations. In an iteration, you still do full lifecycle (requirements, development, deploy), but the Business Analysts are 1 iteration ahead of the developers and the developers are 1 iteration ahead of the deployment team. In Rhythm, there's an Iteration Advocate who gathers and manages requirements, Developers do the development, and the QA team does automation and deployment. I've worked with this system in the past with Brian and this process works quite well.

Iteration Advocate: Traditional "Process Mentor", in disguise. They're responsible for making sure the spin happens and the work gets done. Responsible for one thing: Ensuring the successful delivery of each and every iteration. A big reason that Rhythm advocates this role is because they've found that Project Managers tend to spend 80% of their time doing PM work. By having this role, the PM can concentrate on managing the project and doesn't have to be concerned about managing people and their deliverables. As part of each iteration, artifacts are produced. This makes it possible to put the project in hibernation at any iteration transition point.

Software Iteration Plan (SIP): A simple artifact that includes features and tasks, as well as defects to fix for each iteration. As part of the weekly planning, this artifact is re-visited and updated. Depending on how formal your organization is, this document can be kept on a wiki site, in a word document, etc. Meetings are held every Wednesday afternoon - where PMs sit in a room and team members come in and report their status.

Iteration Transition Meeting (ITM): Backbone of the process - happens every Thursday morning and involves all parties involved in the project. Similar to SCRUM's Post-Sprint Meeting. It's an official buyer/seller marketplace for cross-functioning teams to buy/sell iteration artifacts.

• Definers are selling their requirements, analysis and design specs
• Developers are selling their functionality
• Testing are selling their test plans and specs

Participation requires a minimum of one representative from each respective group. Project Manager conducts, Iteration Advocate facilitates. Brian says this meeting has virtually eliminated the need to do status meetings.

From working with Brian, I know that one of the key ingredients to this process is the Testing Team. The folks that typically fill the QA Developer role is on a whole new level from most testers I've seen. Rhythm Testers are automation experts and know how to setup CruiseControl, run Ant/Maven/Unit tests, as well and configure and deploy to production servers. These folks are the ones who handle the initial installation and configuration of the test, development and production servers.

As part of Rhythm, it's also important to plan for "Featureless iterations" - where nothing is done but run performance tests and regression tests. Done at both 40% and 60% project completion. This was a good presentation about an excellent process that I still try to use today. In my experience, the hardest part to implementing it is finding good QA Engineers that are automation experts as well. Nice job Brian!

 » Download Presentation (PPT)

Lajos Moczar: Creating A Successful Open Source Strategy

Lajos is a committer on OpenEJB and has written two books: one on Cocoon and one on Tomcat. Lajos has been working with open source since 1995 and has a company called Galatea Information Strategies in Colorado Springs. Lajos mentioned that he recently became well-known in the open-source world by writing a paper titled "Start developing and deploying J2EE apps on the first open source J2EE server". This title apparently pissed off the JBoss guys to no end, their PR Firm called JavaWorld and they renamed it to A first look at Apache Geronimo. Because of the JBoss controversy, Lajos became inspired to write The open source monopoly, which explains why he thinks doesn't think JBoss is open-source.

NOTE: The big reason Lajos doesn't like corporate-backed open-source is because he feels that quality app servers like JOnAS deserve lots of press too. However, they don't have the marketing dollars that JBoss has, and are therefore overshadowed. He also mentioned how dangerous it can be when a project's committers get so full of themselves that they start veering from a standard (i.e. J2EE) and developing their own add-ons. If companies and developers use these add-ons, it leads to Vendor Lock-in, which is the same as the lock-in you find in commercial products with proprietary APIs.

Bottom Line for open source adoption: Open Source is mature for the enterprise, as long as you're mature in your approach to it.

The state of OSS: Over the past past 4-5 years, OSS has gone from youth to adulthood. Now a viable choice for large-scale organizations, common talk in business and news. Recent report from CSC on enterprise usages of OSS:

• 70% have Linux and/or Apache
• 42% are using Tomcat
• 14% are using PHP

OSS in young adulthood, where childhood was free software, not a thread to the mainstream. OSS was born of the hacker culture of the 70s (and earlier). Richard Stallman and GNU dominated open source in the 80s and early 90s.

In Adolescence, open source went a big crazy (late 90s). The Big Split was when open source was redefined as a methodology instead of a culture. Eric Raymond, Netscape and the launch of "commercial" open source. Open source != free software. Open source + the Internet = Linux, which leads to popular access to open source.

Now in Young Adulthood, where there is an explosion of choices (in Operation Systems, Containers, Technologies (i.e. PHP, Perl, Python, etc.), Web Frameworks). 34,000 open-source projects on freshmeat, 86,000 on sourceforge. There's also an explosion of licenses. Explosion of technological advances in web services, Java and development technologies. This has all resulted in an explosion of adoption in many large organizations.

Is OSS Mature? Open source faces a crisis of maturity:

• Does it achieve the ideals of its origins and stay as a separate force in the technology world?
• Does it get subsumed by the increasing trend of corporate open source?

With the "corporate open source" model, the project is essentially a product, except that it's free and and you can see the source. Are we losing the intrigue and excitement that the 2:00 a.m. developer experiences? The "corporate open source" model seems to discount the roots of open source - when developers stayed up until the wee hours of the morning to develop something they were passionate about. With the corporate model, it becomes their job and could lead to a loss of passion.

Some Challenges: As OSS matures, there are challenges that face OSS implementers, particularly those in advocacy/evangelization roles. Inherent problems - a lot of releases (difficult to keep up), is it really free, etc.

More Challenges: Overwhelming choices (choose a languages (Java, Perl, C#, Python, C++) , choose a direction w/in a language (i.e. Java 1.4 or 1.5) , choosing a technology direction (J2EE or just part of it - sans EJB).

After this overview, Lajos went into an example of how open source is typically adopted in an enterprise. It starts off with a web application pilot, where a developer downloads Apache + Tomcat, spends a day getting them talking to each other, and then configures everything using the default settings. From there, an application is developed w/o code or directory structure standards, without security and without failover solutions or backups. Nothing is documented as development ensues. This pilot becomes a production application and the IT organization has no idea how to support the application, and new developers have no idea how to maintain it, or start a new project based on the same architecture and tools. Bottom line: using a pilot like this is an irresponsible way to introduce OSS to your enterprise.

Typical OSS Introduction problems: The first step towards successful OS is organizing the problems or "challenges". Use of technology problems, process problems and choice problems. OSS has the strange effect of highlighting inherent problems in your enterprise or IT strategy. These problems can be fixed.

• Technology problems: Technologies come and go, but the core business stays the same. What is important is that you know how to use and what you choose. Commons IT mistakes: The latest and greatest, bleeding edge software, technology for the sake of technology, instead of for the business. Sounds obvious, but you must always start with one question: what is the purpose of IT?
• Process problems: OSS highlights already existing process problems b/c it's very accessible, tends towards permutations in options and versions and there's less work done in the way of best practices than we need. You need a solid infrastructure - processes and principles - to handle the problems with OSS and to insure complete success with it. More than ever, this is the time for process: no OSS implementation should be w/o them! Develop, test, upgrade, deploy, etc.
• Choice problems: Factors influencing choices: technology direction, project viability and health, knowledge base, license type, corporate sponsorship, "spirit" of founders.

Principles of a Pilot: Your pilot is the most critical step in OSS implementation. The pilot must demonstrate: sound and viable technology choices, solid processes and principles, vertical slices.

Starting a web pilot right: Research the options: which version of Apache/Tomcat/MySQL is the most stable (use mailing list). What is the proven integration solution, security "best practices", licensing concerns? Are there documentation concerns? Do you have to pay for it? Is it any good?

When starting with open-source you need to plan, plan, plan.

• Organize your directories for repeatability, ease of maintenance.
• Separate program directories from development directories from deployment directories
• Document all configurables, installation quirks, directory locations
• Define a solid and repeatable build process
• Never take all the defaults! Determine what to change.

Authentication and Security. Problem: multiple logins is the curse of the modern enterprise - don't reinvent! Use single sign-on if possible. Don't have an LDAP server, install one! Kerberos is a good option if you are starting out: a bit of work, but worth it. Problem: OSS is not inherently more secure than anything else. Solution: research for the most low-risk products. Keep currently with he project bug lists and security announcements. Subscribe to all relevant mailing lists.

Development processes. Problem: Need a development process for OSS-based applications. Solution: Use CVS with replication (CVSUP). Use Ant/Maven with a well-documented target/properties schema for builds. Implement a three-stage process, of development, QA and production. Create a simple process for tracking application releases and enabling rollbacks.

While listening to Lajos talk about all of this stuff, I couldn't help thinking to myself that AppFuse solves a lot of these problems for you.

Understand the Pitfalls of OSS:

Version mania and product instability: establish your own release criteria and testing plans. Always upgrade with a rollback plan. Once you find stability, keep it!

Lack of product infrastructure: Great products are useless w/o the proper infrastructure. If any of the following don't exist around an open source project, it's a warning sign that it might not be ready for prime-time.

• premier documentation (i.e. books)
• training services
• tiered support services
• integration services
• security alerts
• intuitive administration and operator interfaces
• interoperability testing - does the release announcement provide upgrade paths?
• load testing
• release criteria (i.e. basic interoperability testing, tested functionality, etc.)

Free software is not really free - there will be costs:

• Training
• Consulting
• Hardware
• Support

Flavor-of-the-month technologies: ask the hard questions about any technology, jumping on the bandwagon for its own sake will be expensive, older proven technologies are often better. Ruby on Rails fits into this category.

Corporate open source and branded technologies: beware brand names backed by VC funding - they could end up being monopolies. Can lead to Vendor Lock-in. Beware of technologies dominated by a single entity. Standards are the best way to prevent this from happening. The best solution: community-driven standards - complete with best practices.

Open source or "free" software requires the same approach and dedication as does commercial software:

• Software infrastructure
• Training, documentation, support
• Sound implementation, upgrade, support processes

At this point, Lajos gave his recommendations of good open-source projects that are ready for enterprise adoption. I wasn't fast enough to write down the ones he mentioned for Security, Integration and End-user applications. He did say he'd post his slides, so hopefully I can fill this list in later.

• Operation System: RedHat/Fedora, SuSE, FreeBSD, OpenBSD, Mandrake
• Data: MySQL (note the license), PostgreSQL, MaxDB, dbXML, Xindice, eXist, Castor, Hibernate
• Security
• Web Services: Apache SOAP/AXIS, Tomcat, Jetty, JBoss, JOnAS, OpenEJB, Resin, Geronimo, LUkAS, ebXML, Jabber, SIM, CryptIM, Magpie (PHP RSS), Jena, ircd, OpenCMS
• Integration: JMS (OpenJMS, SwiftMQ)
• End-user: Firefox, Open Office

Implement: Once you have organized your choices, divide and conquer. Determine what you will keep commercial: mainframe apps? High-end database platforms? Take one slice at a time: database, communications, server platforms, end-user platforms, web services, networking, etc. Be careful of tightly integrated components.

This marked the end of Lajos's (very enjoyable) presentation. I've known of Lajos through his Apache+Tomcat FlashGuides, so it was very cool to meet the man behind the knowledge. It's also pretty neat that we're both doing the same line of work. I hope to sit down and have a beer with Lajos in the near future. During the QA Session at the end, I picked up a couple of audience tips: Google has their open-source code at code.google.com and Revolution OS is a good movie to rent.

 » Download Presentation (PDF)

This is one of the best Denver JUG meetings I've been to in a long time, and this summer looks like it'll be packed full of good sessions. July's JUG Meeting is going to be awesome: Bruce Tate talking about Beyond Java followed by David Geary talking about Ruby on Rails.

AppFuse Videos

I know I said I'd never do an AppFuse video, but after having many requests - I decided to go ahead and make a couple. The first one is a demo of creating a new project and then installing and browsing that project in your browser - to see all the out-of-the-box features.

• http://demo.appfuse.org/setup.html

The 2nd one basically all the stuff that's done in the tutorials - using Spring MVC for the web framework. I create a Person.java object and then use AppGen to generate all the code for it. In this one, I make a number of mistakes (but solve them all). I thought about going fully happy-path, but then decided it was important to show some gotchas that might occur.

• http://demo.appfuse.org/appgen.html

I used the trial version of Camtasia Studio to create these videos. Thanks to Keith at KGB Internet for hosting the demo site for AppFuse. If you need Tomcat hosting, Keith offers an excellent service at a very good price.

Update: You can also download these videos for off-line use.

Update 2: I updated these videos for AppFuse 1.9.3.

JSF needs better tools

In general, I don't like the fact that JSF is designed for tools vendors. However, after seeing a Visual Studio .NET 2005 demo - I can understand why that's Sun's motivation. Visual Studio is *very* cool and seems to greatly simplify ASP.NET development. That's why it's disturbing to see Why do JSF tools suck so bad?.

If the JSF Tools are going to suck (compared to Visual Studio), why don't we just make it more developer-friendly (instead of being so tools-friendly)? Of course, the better solution is to make the tools better, but that doesn't seem to be happening. Maybe we should just try to get Visual Studio to support JSF.

Round Squares with CSS

Want to create squares with rounded corners - and only use CSS? If so, you might want to checkout the CSS Rounded Box Generator. Source: CSS Beauty.

Rails is 8 times slower than Spring+Hibernate

Might as well start off this week by getting people's blood boiling. According to a comment on Dion's blog:

Having done extensive performance (scalability means different things to different people) testing on both Rails and a comparable Spring/Hibernate/JSP2 webapp (no one seems have have done any sort of benchmarking on Rails, or they simply don't care, I don't really know, but since benchmarking is what I do... ) I can say that Apache2/FastCGI/Rails is about... 8x slower than the comparable Tomcat/Spring/Hibernate/JSP2 solution. And that is with caching turned on in Rails (using Rails 0.9.5...)

Quite frankly, 62 req/s on a Dual Opteron with 4GB of RAM rendering a simple view with no DB access is too... damn slow.

Personally, I still think Rails looks like a great (and easy) way to develop webapps. I just wonder if there's some truth to the "can't scale" argument. I guess the best way to find out is for me to develop an application like AppFuse with Rails, and then hammer it (and AppFuse) with JMeter to see what kind of results I get.

On a sidenote, I wonder when Rails will hit the illustrious version 1.0? They released 0.1 last week - which is a bad version number for marketing. If it's as mature as folks claim, why not make the next release 1.0? That version number alone will likely allow developers to use it more in big companies.

Good Saturday

Yesterday was quite possibly the most action-packed days of my life. It all started with going out to breakfast with the family. From there we went on a Hammond Candy factory tour with the Colorado Bus Club. Abbie had a great time and couldn't stop running around in circles. Here's some pictures from the event. After that, I headed to a local connectionless coffee shop and worked on Spring Live for two hours. From there, Jack and I headed to DU Basketball game with some free tickets I got from a friend. That game ended about an hour before the DU Hockey game. After the hockey game, I went and watched a friend's Jazz band play at The Hornet. Phew! It was a great day with a lot of family time. Being a Dad has got to be the coolest thing that's ever happened to me.

Tapestry by Example with Erik Hatcher

It's Saturday morning and I plan on learning more about Tapestry this morning. There's pretty light attendence in the room. Too bad - I bet if it was a JSF talk, the room would be packed. In this talk, we're actually going to build a real application on-the-fly.

Why another web framework?

• HTML should be HTML
• JavaScript Embraced
• Reusable components
• Free developer from dealing with plumbing
  • Request/Response
  • Session/Application Scope
  • URL Mappings (nice, but leads to ugly URLs)
• Event handler driven
• Asset handling
• Line-precise error reporting

Erik says its one of the few frameworks that embraces JavaScript so much. I dig this b/c I think JavaScript is important for web development. Look at what the GMail guys have done with JavaScript. It'd be wicked cool to have this sort of thing open-sourced so we could all create GMail-type interfaces.

Dirty Laundry

• Heavy Duty - can be overkill for small/simple app
• Potentially steep learning curve. More of an un-learning curve. The Servlet API isn't in your face - so all that knowledge is useless.
• Not widely accepted - managers haven't heard of it
• Some awkward conventions (i.e. abstract classes)
  • and some non-intuitive necessary tricks
• Unit testing tough - do you unit test your swing app's listener methods?
  • Tapestry Test Assist should solve this problem
  • Howard's caught the TDD bug, so this should get better
• Lousy URLs - there's a patch on the wiki to fix this
  • Tough to do hit/stat tracking
  J2EE declarative security
• Example applications
  • Workbench and VLib included with Tapestry
  • Better Pet Shop

Workbench Demo - DatePicker component does do i18n (very cool!). Client-side validation does one-field at a time, like WebWork does. I prefer the "all messages in one pop-up" approach that Commons Validator has. Pallete component looks very cool - you can move and re-order items from a multi-select on the left to a multi-select on the right. Chart component looks pretty cool - you can generate graphs very easily. Unfortunately, it's not part of Tapestry, but you can see how to do it in the Workbench app. If you want to see an online version of the workbench, I have it setup on my server.

Repositories for components: Tacos, Tassel, T-Deli and a few mentioned on the wiki.

To bring a component to life, you simply add a "jwcid" to an HTML tag. You can specify initial values for page properties using <property-specification initial-value=""> tag in your template's .page file. The value is implicitly an OGNL expression, and you can define lists using "{'value1', 'value2', 'value3'}". This is great for prototyping before you have a backend or even a page class. To remove elements in a page, specify jwcid="$remove$" on an HTML element and it'll be removed at render time. The restriction is you can't have Tapestry components inside a $remove$ component.

If you don't want to use abstract methods (and hence classes) in your page class, you can use getProperty()/setProperty() methods. However, the recommended way is to use abstract methods.

Templates - two different types. You can put the @component stuff directly into your page - or you can use jwcid's and refer to a name that's defined as a <component> in your page-specification XML file. The Border component can be used to do page decoration like SiteMesh. You can use the Shell component to declare stylesheets/scripts on a per-page basis. This is something I do a fair amount, so it's nice to see it's built into Tapestry.

Internationalization - Resource bundles are component specific (one .properties per page). In a .page, you use <message-binding>. In a template (.html), you use attribute="message:key" or <span key="">. The "key" attribute can't be used on any ol' HTML element, only on the <span> tag. In .java, you use getMessage() and format(). You can also define a custom message source (i.e. read messages from a database). I'm impressed with how easy it is to do i18n in Tapestry. It's also cool that i18n is built-in for templates. Just include a locale extension on your page and it'll be rendered for users with that locale. For example, home_fr.html will show up for users with the French Locale.

Engine - gets all requests. Visit class - POJO that acts as like an HttpSession. You can configure it in the .application file. You can talk to your "Visit" class in a template using "ognl:visit". To call methods on it, just use "ognl:visit.method". Majority of services originate in the Engine. Generally override createXXX methods. Engines can be pooled or created on a per-session basis.

If you override the createXXX methods in Engine, you change the behavior for:

• message source
• global and visit
• property configuration
• template and component sources

contrib:Table - to override use <tr jwcid="columnColumnValue@Block"> - where "column" is the name of the column. Using this, you can easily put links and such into a table cell. Built-in TableModel can be used to talk directly to a JDBC DataSource. The TableModel is smart in that it only brings back the rows it needs to display. Add an exclamation point to the beginning of a column name to turn off sorting for that column. i18n is built-in for headers - the name of the column is simply looked up as a key in the page's .properties file.

Exception handling - Bail out by throwing an ApplicationRuntimeException. This tosses you to the default exception page, which you can override and "pretty up".

Validation - it's robust, but it can only validate <input type="text">. Erik thinks that Tapestry's validation framework could be much simpler and easier to use.

Dynamic Templates - can implement a IDelegateSourceTemplate and pull templates from a database or content-management system. To register your new TemplateDelegate, you can just register an <extension> in the .application file and point it to your class.

Page Lifecycle - initialize(), PageRenderListener(), PageValidateListener(), PageDetachListener(). Can use an ExternalCallback and ICallback to set properties on one page from another. Callbacks look very cool and there's a lot of discussions about them on the mailing list. The VLib app has a fair amount of callback examples.

This was definitely a good Tapestry session - thanks Erik.

AppFuse's License

Someone sent me an e-mail today and made me aware that the Commons Attribute License that AppFuse uses only applies to documents and such:

"Collective Work" means a work, such as a periodical issue, anthology or encyclopedia, in which the Work in its entirety in unmodified form, along with a number of other contributions, constituting separate and independent works in themselves, are assembled into a collective whole.

Because of this, I'm thinking of switching to use an Apache 2 License. This seems to be the friendliest license for open source. Basically, all I want to do is try to get folks that use AppFuse to give it some credit. Which really means they're marketing it in a sense. More marketing -> more users -> (hopefully) I'll stay employed and the product will get better.

The only pain I can see about licensing with an Apache License is they want you to add a short copyright notice at the top. But I'm not really handing the copyright of AppFuse over to ASF, am I? Do I have to add this copyright to all my .java files? I know that Spring does this, but... ugh ... seems rather unnecessary.

Java Developers - let's take over the MySQL Conference!

Last April, I attended the MySQL Conference in Orlando. It was a good time, but the number of Java Developers there was pitiful. I had a good time hanging out with Mark (Matthews) and Anthony, but it was still a poor showing. Now MySQL is gearing up for this year's conference (April in Santa Clara) and I'm thinking about submitting a proposal. Anyone interested in joining me? Let's bombard them with proposals and see if we can take over the sucker! If nothing else, it'd be cool to hang with more Java Developers in California.

The conference actually ends the day before Julie and I's 5th year anniversary (April 22nd). If my proposal gets accepted - I think we're going to have Julie's sister babysit and Julie will fly out to spend the weekend touring around wine country. Sounds like fun, eh? Now I just need to find some sort of limo so we can do lots of tasting.

This site crashes a lot

It's plagued me before, and now it's back again. When I go to sleep at night, this site crashes. I've been up until 4 lately and it still seems to wait until I fall asleep. In fact, my 500 page is my top referrer this month. That's not good. The problem is probably not Roller - since JRoller seems to stay up and humming. Although it does die, I do have somewhere around 30 open database connections.

The truth is I need to run a profiler and figure out what's wrong. Unfortunately, I don't have the time and it doesn't really bother me that much. I just end up with a whole bunch of "your site is down" e-mails every morning when I wake up. Hopefully I can fix it soon. I've been telling people I'll have more time when the baby comes. They think I'm kidding - but I think it's true. My job will be handling Abbie while Julie nurtures the new baby. Abbie sleeps 12 hours at night, and 2-3 hours in the afternoon. I sleep 4-5 hours a night. That's a lot of time to get stuff done.