Raible Designs

The Angular Mini-Book 3.0 is now available!

Matt Raible — Fri, 23 Jun 2023 05:44:05 -0600

I'm excited to announce that the Angular Mini-Book 3.0 is now available! You can download it in PDF and EPUB formats from InfoQ.

About this book

Spring Boot is a popular framework for building REST APIs. You'll learn how to integrate Angular with Spring Boot and use security best practices like HTTPS and a content security policy. What about deploying your Angular app to the cloud? Yep, it covers that too!

What's new?

This new edition uses Angular 15 and Spring Boot 3.0. Some other fun facts:

Tech reviewed by Angular expert Alisa Duncan. She improved many of the code samples, and I'm grateful for her help. You should follow her on Twitter!
First commit on 3.0 version: Feb 16, 2023
Files changed since 2.0: 558
Build date: June 9, 2023

You might say this book is outdated on launch. However, I created a GitHub Action to upgrade all the examples to use Angular 16 and Spring Boot 3.1. This action proves all the code should work if you create apps with the latest Angular and Spring Boot versions. 😊

To send us feedback or issues, e-mail InfoQ at feedback@infoq.com, email me at matt@raibledesigns.com, or hit me up on Twitter @mraible.

The Angular Mini-Book 2.0 is now available!

Matt Raible — Fri, 18 Feb 2022 08:24:39 -0700

I'm pumped to announce that the Angular Mini-Book 2.0 has been released! You can download it in PDF and EPUB formats from InfoQ.

About this book

What's new?

This new edition (2.0) uses Angular 13 and Spring Boot 2.6. Some other fun facts:

First commit on 2.0 version: Nov 19, 2021 { Thanks to James Ward for his help upgrading to Knative 1.0! }
Upgraded the book's project to use Gradle 7: Dec 2, 2021 { I appreciate you Guillaume Grossetie! }
Upgraded to Angular 13 and Spring Boot 2.6.1: Jan 7, 2022
Detailed QA of all the code: Jan 27 - Feb 10, 2022
Files changed since 1.0: 244
Build date: Feb 17, 2022

For more information about this book, please read my post about its 1.0 release.

To send us feedback or issues, e-mail InfoQ at feedback@infoq.com, email me at matt@raibledesigns.com, or hit me up on Twitter @mraible.

The Angular Mini-Book 1.0 is now available!

Matt Raible — Wed, 17 Nov 2021 14:48:40 -0700

I'm pleased to announce that the Angular Mini-Book has been released! You can download it in PDF and EPUB formats from InfoQ.

About this book

The Angular Mini-Book is a guide to getting started with Angular. You'll learn how to develop a bare-bones application, test it, and deploy it. Then you'll move on to adding Bootstrap, Angular Material, continuous integration, and authentication. Spring Boot is a popular framework for building REST APIs. You'll learn how to integrate Angular with Spring Boot and use security best practices like HTTPS and a content security policy.

This initial edition (v1.0) uses Angular 12 and Spring Boot 2.5. I do plan on updating it for Angular 13 and Spring Boot 2.6. If you have any tips for upgrading, please let me know!

Purpose of the book

I think building web and mobile applications with Angular, Bootstrap, and Spring Boot is a great experience. I'd like to encourage more developers to try it.

Thanks!

I'm incredibly grateful to Trish, Abbie, and Jack. They put up with my late nights and extended screen time while I worked on this book.

To Miško Hevery, thank you for inventing Angular and changing the lives of frontend developers. To the whole Angular team, I appreciate you and all you do for the community. Kudos for making our apps faster with every release!

To Phil Webb and Dave Syer, thanks for creating Spring Boot and simplifying Java for everyone. Hats off to the whole Spring team for their tireless dedication to quality open source projects.

I want to thank this book's tech editor, Deepu K Sasidharan. I looked to him for his deep experience with both TypeScript and JavaScript. Many sections are more streamlined because of his advice.

This book's copy editor, Maureen Spencer, helped correct my grammar and make this book easier on the eyes. I'm thankful for your help, Maureen.

David Neal created the cover image and all of the illustrations. He's awesome! You should consider buying his stuff and follow him for his dad jokes.

Finally, my compliments to you, potential reader. It's a heckuva time to be writing code. Enjoy your learning adventures! 😃

Fun facts

Creating this book was more difficult than I imagined. As you can see from the timeline below, I didn't do a whole lot in 2020. I ramped things up this spring and tried to get 'er done before Angular 13 and Spring Boot 2.6 were released. At the very least, I succeeded on the second goal, even if only by a couple days.

First commit: June 25, 2020
Outline created: September 15, 2020
Started writing: October 3, 2020
Upgraded to Angular 12 and Spring Boot 2.4.1: May 4, 2021
Finished first draft: June 30, 2021
Build date: November 16, 2021
Total commits: 108

This book was authored with Asciidoctor and I appreciate the assistance I received from Guillaume Grossetie.

The source code repo contains all the demos in addition to the book's code. The Count Lines of Code report (excluding generated files and directories) shows that it's mostly words, followed by example code:

--------------------------------------------------------------------------------
Language                      files          blank        comment           code
--------------------------------------------------------------------------------
AsciiDoc                         14           1651             20           5312
HTML                             38            211              0           2349
TypeScript                       86            322            299           2057
JSON                             27              0              0           1346
CSS                              11            265            250           1288
YAML                             10              8             89            654
Kotlin                           18             87              4            499
JavaScript                       12             35             48            355
Gradle                            6             43              4            227
Bourne Shell                      2             50            144            225

I'm proud to have built the v1.0 version of the book on my parent's 49th anniversary. Happy Anniversary Mom and Dad!! 🥳

To send us feedback or issues, e-mail InfoQ at feedback@infoq.com, email me at matt@raibledesigns.com, or hit me up on Twitter @mraible.

Announcing Angular CRUD 2.0

Matt Raible — Mon, 27 Sep 2021 16:29:15 -0600

Friends shouldn't let friends write CRUD apps. At least, not by hand. That's why I used a schematic called Angular CRUD in one of my last Angular + Spring Boot blog posts. That post is kinda old (January 2020), but the combination of Angular and Spring Boot remains popular. That's why I decided to turn the series into an Angular Mini-Book for InfoQ.

The book has five different sections:

Build an Angular App
Integrate Angular with Spring Boot
Beautiful Angular Apps with Bootstrap
Angular Deployment
Angular and Docker

My goal with the book is to show you everything you need to get your Angular + Spring Boot app to production. But, what about JHipster?

Of course, you can just use JHipster, but I've found that a lot of beginners are intimidated by all the code it generates. That's why I wanted to create a bare-bones Angular guide that uses Spring Boot for its API.

I published the news on Twitter at the end of June.

I'm happy to announce that I've finished the first draft of the @Angular Mini-Book! After the editing process, you'll be able to find it at a local @InfoQ website.

Hopefully, they'll create a better cover for it. 😅 #angular #springboot #asciidoctor pic.twitter.com/el20GCS24s
— Matt Raible (@mraible) July 1, 2021

As part of creating the book, I updated Manfred Styer's Angular CRUD project and released v2.0. The 2.0 release adds support for Bootstrap and Angular Material for CSS framework aficionados, like me.

You can see the getting started tutorial for how to use it in your projects. To inspire you, I've included some screenshots of the new Bootstrap and Angular Material support.

Angular CRUD + Bootstrap

Angular CRUD + Angular Material

This theme is based on Create a responsive sidebar menu with Angular Material by Zoaib Khan.

Introduction to Angular CRUD Video

I also published an Introduction to Angular CRUD video that shows how it works.

You might ask, "When will the Angular Mini-Book be published?"

We're about halfway through tech editing. I hope to finish that in the next month, pass it on to copy editing, and publish before the end of the year.

Keep an eye on InfoQ's Mini-Books and follow me @mraible on Twitter for updates.

Life as an Open Source Developer, One Year Later

Matt Raible — Mon, 6 Nov 2017 08:33:17 -0700

It's been a little over a year since I wrote about life as an open source developer. I'm happy to say I still haven't written a single line of proprietary code. Of course, things have changed a lot in the last year. I thought going full-time would bring stability to my career. Instead, six months into it we joined forces with Okta.

The transition was rough at first. At Stormpath, we had full-featured SDKs and a great relationship with developers that used our service. We were able to port many of our SDKs to work with Okta, but we discovered that Okta didn't have a great relationship with developers. In fact, their developer blog hadn't been updated in over a year when we arrived.

On the upside, Okta's API supported standards like SAML, OAuth, and OpenID Connect. Open standards made it possible to use other frameworks and not have to rely on our own. I was pumped to find that Spring Security made it easy to integrate with SAML and OAuth. In fact, I was able to leverage these standards to add OIDC support to JHipster.

Okta's new developer console and open pricing are just a couple examples of improved happenings since we arrived. The Okta Spring Boot Starter and JavaScript libraries for Node.js, Angular, and React are also pretty awesome.

I'm happy to say my contributions on GitHub almost doubled in the last year!

As far as stress is concerned, that hasn't changed much. I've learned that the stress I feel from work is still causing me to have high blood pressure. When I measure it in the mornings, or at night, it's fine. When I measure it during the day, it's elevated. I believe my high blood pressure is caused by doing too much. Sure, it's great to be productive and accomplish a lot for my company, but it's killing me.

Therein lies the rub. I get to create my job. All I'm asked to do is write a blog post per week and speak at a conference (or meetup) once a month. Yet I'm doing way more than that. Since this time last year, I've delivered 33 presentations, in 13 different cities. I keep a page on this blog updated with all my presentations.

Next year, I still plan to speak a lot, but I plan on toning things down a bit. I'll be concentrating on US cities, with large Java user groups, and I'll be limiting my travel overseas.

Outside of my health concerns, I'm still loving my job. The fact that I get paid to speak at great conferences, write example applications, and discover new ways to do things is awesome. It's also pretty sweet that I was able to update the JHipster Mini-Book and upgrade 21-Points Health during work hours. The fact that I got featured on the main Okta blog was pretty cool too.

The good news is my overseas travel isn't done this year. Today, I leave for Devoxx Belgium, one of my favorite conferences. It'll be my first time in Antwerp without Trish. However, I'm speaking with friends Josh Long and Deepu Sasidharan, so it's sure to be a good time. Traveling to Devoxx Morocco should be fun too. I've never been to Casablanca before.

In December, you can catch me at SpringOne and The Rich Web Experience. Next year, I'll be speaking at Denver Microservices meetup, Utah JUG, Seattle JUG, and JazzCon. I plan to do a JUG tour in the northeast US too.

You might've noticed I don't write a lot of technical content here anymore. That's because I'm doing most of my writing on developer.okta.com/blog. I'm still writing for InfoQ as well. I really enjoyed attending the JavaOne keynotes and writing up what I saw.

I'll leave you with this, a project I'm working on actively and plan to finish before Devoxx Morocco.

Had a good hacking session today w/ @java_hipster & @Ionicframework. Creating a JHipster-enabled Ionic client works! https://t.co/5biHQDO941
— Matt Raible (@mraible) November 3, 2017

Viva la Open Source!

Life as an Open Source Developer

Matt Raible — Thu, 3 Nov 2016 16:29:01 -0600

It's been a little over a month since I started my new gig at Stormpath. I gotta say, life is great as an open source developer! Yes, I did start working for them as a consultant in April, so it's not a huge change for me. However, I only recently realized I haven't written a single line of proprietary code the entire time. My GitHub contributions look pretty good this year. They're nothing like @mojavelinux, or @dsyer, but I'll get there.

It's also been a bit more stress than I'm used to. I think this comes from a couple things: 1) turning my hobby into my job and 2) I've set a lot of high expectations for myself. As a developer evangelist, I get to create my own job. That means I can speak at the conferences I want to, write the code I want to, create the blog posts I want to, and everything else in between.

At the end of September, I finished updating the JHipster Mini-Book for JHipster 3.x. It's gone through tech editing and it's being copy-edited right now. I hope to release it within a week.

In early October, I said I'd commit to writing one blog post per week, develop a JHipster module for Stormpath, and help get their Angular 2 support good enough for an alpha release. I'm happy to report I've been able to accomplish most of these and I hope to show off our Angular 2 support soon.

I then channeled my efforts into integrating Stormpath's Java SDK with their AngularJS directives. You can read about how I did that in Get Started with AngularJS, Spring Boot, and Stormpath. Unlike my previous AngularJS tutorial, this one connects to a backend and shows how to communicate with Spring Boot cross-domain.

If you like to read code more than words, you can look at the example project's commits on GitHub.

Create an AngularJS UI: search and edit features
Create a Spring Boot app with Stormpath: app from start.stormpath.io
Develop an API to CRUD people with Spring Data REST: /api/people
Integrate AngularJS and Spring Boot apps: cross-domain
Integrate Stormpath into AngularJS for login, registration and forgot password: Stormpath Angular SDK

Last week, I released a JHipster module that integrates Stormpath. This exercise was good because I was able to identify some gaps in Stormpath's SDKs and fix them. Getting something to work made me feel good; having the ability to improve the developer experience was even better! Of course, I blogged about what I learned.

This week, I edited and code reviewed some posts from Karl Penzhorn on React with Spring Boot and using webpack with React. I also got to bang my head against the wall writing Angular 2 tests. If you're writing a module for Angular 2, generator-angular2-module provides a nice starting point.

Last, but certainly not least, I'll be speaking at a few events about Microservices, JHipster, Angular 2 and Stormpath in the near feature.

Rocky Mountain Software Symposium, November 19
The Rich Web Experience, December 9
A joint talk at Denver JUG with the infamous Josh Long, December 14

If you have any questions about developer evangelism, the technologies I mentioned in this post, or Stormpath, please let me know. Otherwise, I hope to see you on the road soon!

RE: Customizing an Asciidoctor PDF so it looks like an InfoQ Mini-Book

Matt Raible — Wed, 28 Oct 2015 10:41:38 -0600

Last week, I wrote about customizing an Asciidoctor PDF so it looks like an InfoQ Mini-Book. Shortly after writing that blog post, Dan Allen responded to my questions and showed me how to customize Asciidoctor's PDF generation. I ended up using both techniques he described: creating a custom theme and using Ruby to override methods. To recap, here are the changes I was hoping to make:

The colophon is not aligned to the bottom of the page.
The title page (first one after the cover) and colophon pages should be merged.
The dedication and acknowledgement headers are not center-aligned and underlined like InfoQ's format.
The main sections don't have whole-page delimiters.
The table of contents comes right after the title page, rather than after the dedication and acknowledgement.

I'm happy to report that I was able to fix most these issues, except for the second one and last one. There is a pull request to allow changing the location of the table of contents, but I was unable to make it work. I spent a good hour building the asciidoctor-pdf gem and trying to modify AsciidoctorJ to use it. In the end, I decided to mark this as a bug in the JHipster book and we'll fix it when Asciidoctor supports moving the table of contents.

To customize the output, I created an src/main/ruby/asciidoctor-pdf-extensions.rb file and added the following code to it:

require 'asciidoctor-pdf' unless defined? ::Asciidoctor::Pdf

module AsciidoctorPdfExtensions

  def layout_title_page doc
      # no title page
  end

  def layout_chapter_title node, title
    if node.id == "dedication" || node.id == "acknowledgements"
      layout_heading_custom title, align: :center
    elsif node.id.include? "mini-book" # colophon
      move_down 470
      layout_heading title, size: @theme.base_font_size
    elsif node.id.include? "jhipster" #chapters
      puts 'Processing ' + node.id + '...'
      move_down 120
      # set Akkurat font for all custom headings
      font 'Akkurat'
      layout_heading 'PART', align: :right, size: 120, color: [91, 54, 8, 13], style: :normal
      move_up 40

      part_number = "ONE"
      if node.id.include? "ui-components"
        part_number = "TWO"
      elsif node.id.include? "api"
        part_number = "THREE"
      end

      layout_heading part_number, align: :right, size: 120, color: [42, 1, 83, 1], style: :bold
      layout_heading title, align: :right, color: [42, 1, 83, 1], style: :normal, size: 30
      move_up 30
      start_new_page
    else
       # delegate to default implementation
       super
    end
  end

  def layout_heading_custom string, opts = {}
      move_down 100
      typeset_text string, calc_line_metrics((opts.delete :line_height) || @theme.heading_line_height), {
          inline_format: true
      }.merge(opts)
      move_up 5
      $i = 0
      underline = ''
      while $i < string.length do
          if string == 'Dedication'
            underline += '/////'
          else
            underline += '//////'
          end
          $i += 1
      end
      if string == 'Dedication'
          underline += '////'
      end
      typeset_text underline, calc_line_metrics((opts.delete :line_height) || @theme.heading_line_height), {
            inline_format: true, color: 'B0B0B0', size: 8, style: :italic
      }.merge(opts)
      move_down 20
  end

end

Asciidoctor::Pdf::Converter.prepend AsciidoctorPdfExtensions

Then I modified build.gradle to use this file.

asciidoctor {
    backends 'html5', 'pdf', 'epub3'
    attributes 'sourcedir': '../../../main/webapp',
            'source-highlighter': 'coderay',
            'imagesdir': './images',
             toc: 'left',
             icons: 'font',
             linkattrs: true,
             encoding: 'utf-8',
            'setanchors': true,
            'idprefix': '',
            'idseparator': '-',
            'docinfo1': 'true'
    requires file('src/main/ruby/asciidoctor-pdf-extensions.rb')
}

After getting this to work, we're very close to publishing the JHipster Mini-Book! Thanks to Dan for creating Asciidoctor and supporting this great open source project. It's been a pleasure to write with it and the editing process with Git and pull requests has been wonderful.

Update: The JHipster Mini-Book has been released!

Customizing an Asciidoctor PDF so it looks like an InfoQ Mini-Book

Matt Raible — Thu, 22 Oct 2015 14:15:21 -0600

Earlier this month, I finished the rough draft of the JHipster Mini-Book. Since then, I've been working with editors to get it ready for production. I've also been working with InfoQ to try and make the generated PDF look like their current mini-books. I wrote the book using Asciidoctor and I'm using Gradle to generate HTML5, PDF and EPUB versions.

After doing some research on Asciidoctor PDF themes I created an issue in the asciidoctor-pdf project. My reason for was to see if it was possible to customize certain sections of the generated PDF. The main issues I've had in making the PDF look like an InfoQ mini-book are the following:

The colophon is not aligned to the bottom of the page.
The title page (first one after the cover) and colophon pages should be merged.
The dedication and acknowledgement headers are not center-aligned and underlined like InfoQ's format.
The main sections don't have whole-page delimiters.
The table of contents comes right after the title page, rather than after the dedication and acknowledgement.

After thinking about this a bit more, I thought of a few possible workarounds.

I could add a number of line breaks at the beginning of the page to push everything down to the bottom.
We could delete the title page (with Preview on a Mac or another PDF editor).
We could create new PDF pages that have InfoQ's headers and my content. Then, using a PDF editor, we could delete pages and put the new ones in their place.
There might be a way to have no text in a section's title (so it doesn't show up at the top of a page) and do the same copy/paste of an InfoQ section-delimiting page (with large Part One text) before the section. The hard part here might be lining up the table of contents with page numbers.
Move pages around in the PDF and renumber pages using a PDF editor.

Even if all these workarounds are possible, this will only work for the PDF. InfoQ has asked me to make similar header customizations for the EPUB/MOBI versions.

I looked at the PDF theming guide and it looks like many things are customizable, but they're global customizations, not per-section customizations. Dedication, Acknowledgement, Preface, and Chapter Titles all live on the same level (level 2). I believe it's possible to customize how they all look, but I haven't figured out how to change an individual title.

The only thing I can think of beyond these workarounds are 1) hiring someone to create a custom theme for InfoQ or 2) forking the project and trying to make customizations to the source code myself.

I haven't had any feedback from the Asciidoctor team, so I'm posting this here to try and reach a wider audience. If you've authored a book in Asciidoctor, did you customize the output to fit your publisher's desired format, or did you just take the out as-is and publish it?

The Passionate Programmer by Chad Fowler

Matt Raible — Sat, 10 Oct 2009 13:59:19 -0600

Today I'm attending Developer Day in Boulder, Colorado. The opening keynote was done by Chad Fowler and below are my notes from his talk.

Chad recently wrote a book called The Passionate Programmer. It's about Creating a Remarkable Career in Software Development. Why is it important to do this? Because the average adult spends 53% of their time working, so it's really about creating a remarkable life. What is remarkable. For Chad, it was driving around the country in a 1986 Tioga motor home for two months. He worked the whole time and got to enjoy a lot of beautiful scenery along the way. He just returned from "working" in Hawaii for the last two weeks.

One of the things that motivated Chad to take control of his career was The Pragmatic Programmer book. It taught him you should always try work with people that are smarter than you. What we all really strive for is freedom, especially creative freedom. The whole idea of a remarkable career is it's different for everyone. Some folks want to get rich, some want to take a lot of vacation, some want to travel the world.

The process of developing a remarkable career is fairly easy - you just need two things. You have to have the intention and a system for getting it done. Most people do their careers by coincidence, but they don't drive it. In the music world, no one gets into it for the paycheck. Everybody gets into it because they think they're going to be the best. When Chad became a programmer, he thought the same thing - that he wanted to be a rock star.

When you have a plan, it makes hard things easy. A good example of a helpful planning technique is training for a running race. In September, Chad completed the Indian Summer Half Marathon and he used a training program that introduced mileage in a systematic way. Because of this, training never felt difficult. Another system that Chad recommends is the Seinfeld Calender, where you have a calendar that you X the days when you completed your training (or steps toward a goal). An interesting open source version of this is Calendar About Nothing.

One good way to look at your career is to think of yourself as a product. Choose your market, invest in yourself, execute and market your skills. You should hang out with people you want to be like and work with people you want to become. Chad recommends not only learning a new language every year, but also learning a new domain. You should decide if you want to be a generalist or a specialist. Being a specialist does not mean only knowing one thing, it just means you know one thing very well. If you do both, it's the best path towards awesomeness.

The thing that most programmers don't do enough of is practice. CodeKata is an example of how you can practice. Don't just practice programming, understand how your business operates. One of the best ways to do this is learn how to read a balance sheet. Don't be a Partial Person. Don't be someone who says I'm not a UI programmer. Learn how to do it. If you say "I've always wanted to", you should do it (unless it's illegal of course).

Execution is a mindset. The best person is not always the smartest person. People who struggle have to come up with systems to make them stronger. People who get things naturally (fitness, brains, etc.) tend to not keep perfecting their skill. Always think about how much you cost per hour and try to do something you can tell your boss everyday. Another way to execute impressively is to do an 8-hour burn (similar to the 40-hour work week)

The best way to market yourself is to be remarkable. When you do networking events, try to help people. We can all get stuck in the trap of saying I am an X programmer. This happens a lot when you're successful at X. Don't limit yourself by tying yourself to one technology. As a programmer, your job doesn't suck. If what you're doing is not fun, then you're probably doing the wrong thing. Ruthlessly cut out the crap you don't enjoy.

Lean Teams: Doing more with less

Matt Raible — Wed, 23 Sep 2009 21:20:28 -0600

This evening I attended the Denver Rails User Group (a.k.a. DeRailed) to hear a presentation by Marty Haught. It was titled "Lean Teams: Doing more with less" and the following are my notes from the event.

Today's talk is about "Rocking with Ramen" - a.k.a. working with less funds to make great things. Lean comes from the manufacturing world in that you should Add Nothing but Value. The most important thing you should do is add business value.

The Seven Wasteful Sins for manufacturing are:

Overproduction
Inventory
Extra Processing Steps
Motion
Defects
Waiting
Transportation

The key to fighting overproduction in software is to trim features to those that achieve the greatest value. You should do "the simplest thing that could possibly work" and delay commitment as long as you can because YAGNI.

A minimum viable product is a starting place for validated learning with the least amount of effort. It should be embarrassing. Early adopters see the potential. Rails Rumble and Startup Weekend are good examples of promoting this type of development.

Unused and useless features are best solved by feedback-driven development. This is a process for validating value and creating software that people use. The end result is that you create software that people use and you're able to pivot your plan as you learn. The benefit of this is you stay humble and you don't drink the Kool-Aid (e.g. VC's tell you you're going to be the next Twitter).

The first part of feedback is "Pirate Metrics" by Dave McClure. The main things to track are acquisition, activation, retention, referral and revenue (AARRRR!). The main things you should gather from metrics is they're actionable and should help you make decisions. Vanity metrics like hits-per-month and such should be ignored.

Other feedback options include net promoter score (popup question to ask if users would recommend to a friend), feedback form (make it easy for users to tell you what you think about your product), A/B testing, and usability testing.

The final point is that it's OK to remove features.

To reduce extra processing and waiting, you should implement "Kanban". It's a pull-based system for a continuous flow of work and can be used in software projects to manage/schedule work for cross functional teams. It's an expression of just-in-time and has an emphasis on flow. It's all about getting across the board as fast as possible. In agile development, this is often expressed as a card-based system on a wall in the same room as your development team. Things can only move from the left-to-right as there is space for them. Marty is showing a screenshot of a "Zen" tool he uses on his projects. It has 3 columns (Definition, Work and Verification) from left-to-right that allows you to easily move stuff.

The most important thing about Kanban is it helps to eliminate constraints. The Zen tool only allows a certain amount of items in the "Work" column and it visually communicates blocked items by moving them to the top and highlighting them with a red border. The Zen tool that Marty is showing looks similar to Rally, but is much more visually appealing.

The benefits of Kanban include:

simple, less process
less inventory of requirements/stories
limit work in progress, maximize throughput
less time in meetings
more naturally represents story lifecycle
more easily spot bottlenecks
estimate only if it adds value

Kanban promotes tracking how long it takes for a story get across the board and into production vs. tracking velocity of a team.

On my current project, we use Rally, a small team and have two week iterations. Because the things that Marty is talking about seem to be things we're already doing, I asked him how Kanban differs from Scrum with small teams. He explained that this biggest difference is Kanban is most useful when you're pushing things to production with each iteration.

The most controversial practice that Marty promotes is Continuous Deployment. This is the automated deployment of code to production. It includes automated testing and continuous integration, simple deployment/rollback scripts, a successful CI build triggers deployment, and there's real-time alerts in production. When shit goes wrong, you should use the "five whys" to perform root cause analysis. Marty admits that this is only a good idea when there's a high-level of trust in your development team and lots of tests to prove nothing is broken.

The benefits of continuous deployment is there's a lower story cycle time, you eliminate waste in deploying code, you deliver features/bugs fixes faster and you find integration issues quicker and in isolation. It's also a great way to promote not checking in shitty code.

The skeptics think this is a bad idea because 1) it's scary, 2) they believe it causes lower quality and 3) it causes more issues in production. The good news is you can still control production deployments with your source control system (e.g. branches and such). More than anything, it forces you to have a high quality continuous integration system that acts as the gatekeeper for what goes to production.

You can learn more about topics Marty covered in this talk at the following sites:

Eric Ries - http://www.startuplessonslearned.com
Steve Blank - The Four Steps to the Epiphany
Mary Poppendieck - Lean Software Development
Dave McClure - http://davemcclure.com
Kanban Applied to Software Development: from Agile to Lean

If you're lucky enough to be attending Aloha on Rails, Marty will be presenting there. I recommend you attend his talk if you're trying to get stuff done quickly and get it into production even quicker. His techniques seem to be invaluable for developers that are trying to maximize their efficiency and reduce the time it takes to get their code into production.

How I recovered data from my failed Linux box

Matt Raible — Thu, 11 Dec 2008 14:02:29 -0700

Yesterday, I decided to quit procrastinating and finish up my 2007 taxes once and for all. When I booted up QuickBooks on my XP box, it said it couldn't connect to drive Q. Drive Q is on my Linux box, which I discovered wasn't on. When I booted it, the screen showed an "Error 18" after the GRUB loading message. The resulted in several hours of grub-install and many other commands to no avail.

Since I hadn't messed with the box in almost a year, I didn't even know if it had Fedora or Suse installed on it. I had both disks lying around, so I tried the good ol' linux rescue with my Fedora disks. I was able to access the data, but had no luck in getting network connectivity or copying files to a USB drive.

Today I hoped for a different route: Live CDs. Yesterday, I discovered I was running Suse 10, so I downloaded a Suse 11 Live CD, burned it and booted. It worked, but I didn't have access to my hard drives and wasn't able to mount anything. Next up, I tried Knoppix, which allowed me to boot and access my hard drives. Unfortunately, I still didn't have any network access and my 2GB thumb drive didn't hold enough data.

Next, I pulled out a 250GB USB drive I had lying around. Knoppix recognized it, but was unable to format it for some reason. I plugged it into my XP box, used fat32format to format the drive as FAT32, and plugged it back into my Linux box. Success! I was able to copy all the data I needed and now I have the USB drive plugged into my Airport Extreme.

Hopefully if someone else runs into similar issues, they can use this post to find their path to success.

Appcelerator with Matt Quinlan

Matt Raible — Tue, 7 Oct 2008 19:08:19 -0600

This evening I attended the Denver Open Source User Group meeting where the Basic Concepts talk was on Appcelerator. Matt Quinlan (Twitter, Blog, LinkedIn) was the presenter. I arrived 10 minutes late, so I didn't hear any of the intro stuff. Below are my notes from the event.

The Appcelerator developers liked the "onclick" syntax, but found it was too limited to do everything they wanted. Rather than onclick, they use an on attribute. For example:

<div on="click then l:show.box">
Click me
<div>

"DOM Manipulation is JavaScript cruft."

Appcelerator allows you to implement the Observer pattern in the browser. In addition to allowing DOM elements to subscribe to messages, server-side objects (in any language) can subscribe as well. In Appcelerator syntax "l:" means local and "r:" means remote. The messages that are passed to the server are JSON and have payloads. JSON is more popular than XML because you can eval() it and create JavaScript objects from it. Appcelerator allows you to do Declarative Ajax. On the server-side, you can annotation Java and C# classes with @Service to subscribe to messages. In other languages (i.e. PHP), Javadoc-style comments are used.

Tagline: The seamless fusion of RIA and SOA.

Web Expression Language
Goal is to eliminate 90% of the JavaScript you write. Example syntax:

on="[condition] then [action]"

Conditions include DOM events (click, focus, blur, change, mouse events), key events (up, down, press), other (history, drag/drop, selected, resize, iPhone orient, sortXYZ), subscribe to custom message. Actions include Scriptaculous effects (show/hide, fade, move, drop, grow), set element value (static, dynamic, bind), set CSS class or attribute, execute custom JavaScript, publish custom message.

Now Matt is doing a demo on http://try.appcelerator.org. This site consists of a form that allows you to type in Appcelerator code and run it. 3 attributes can be added to any tag: draggable, droppable and resizeable.

Client-Side
Add simple tags to your HTML to inject RIA widgets. Add single property to existing HTML elements for dynamic behavior. Eclipse Plugin built on Aptana, but is generally targeted towards web developers moreso than business analysts (no drag and drop of widgets).

Server-Side
Server-side development done with your IDE of choices. Based on your server-side tehchnology platform. Easily create services using annotations.

App Command
The app command is similar to Rails' GEM command. Allows you to build scaffolding and deploy to cloud (AppEngine, Amazon S3). It also allows you to pulldown components from the main server and auto-updating.

Examples:

appTunes: widget that wraps Flex Cover Flow widget
Snapshot: photo editing application
Radio Javan: Persian music online
SkyBlox: wireless company

Appcelerator allows you to create prototypes easily by using a JavaScript file with mocks for the server-side objects. In the next version, you can "annotate" the UI and allow end-users to Ctrl+Click on elements and add feedback. For an example of this, see http://dev.appcelerator.com/pearson.

When starting with Appcelerator, you can start by crawling (including appcelerator.js for widgets) then move to walking (decouple server-side and client-side) and finally running (developing working prototypes with mocks for server-side).

"Let's face it, ASP, JSP, PHP and Ruby are just lipstick on CGI."

Performance Testing Memcached

Matt Raible — Fri, 19 Sep 2008 11:29:33 -0600

Earlier this week, co-workers and I did some performance testing with Memcached. We wanted to see how long it'd take to send different sizes of objects over the wire to Memcached on a remote server.

We setup a simple environment with 2 Mac Pros both running 2 x 2.8 Ghz Quad-Core Intel Xeon processors and 12GB of RAM. We used one machine as a Memcached server and one machine running an application with the addition of a new Servlet Filter. The Servlet Fitler read a size parameter and used it to set the size of the object being written and read from the Memcached server. We used JMeter to put load on the box. The following describes the load and the results.

Write Times
Write tests were performed with a single user executing 1000 sequential remove then writes.

Size in Bytes	Time (ms.)	Min. Time (ms.)	Max. Time (ms.)	Total
20000	1.31	0	7	1000
50000	1.834	1	8	1000
100000	2.87	2	9	1000
500000	12.641	9	283	1000

Read Times
Read tests were performed using 50 users with 1 sec. ramp times executing 100 reads each.

Size in Bytes	Time (ms.)	Min. Time (ms.)	Max. Time (ms.)	Total
20000	4.8414	1	375	5000
50000	18.343	1	354	5000
100000	46.181	2	415	5000
500000	137.7328	6	953	5000

During our tests, Memcached was started using the following settings:

memcached -d -m 2048 -M -p 10171 -vv

If you've done similar performance testing with Memcached, we'd love to see your results.

Presenting Web Frameworks of the Future Tomorrow in Denver

Matt Raible — Wed, 30 Jul 2008 21:56:17 -0600

Tomorrow (Thursday) night, I'll be doing an encore presentation of my Web Frameworks of the Future at DeRailed. If you're in Denver and would like to hear me ramble while drinking a beer, join us at Forestroom 5 at 6:30.

After the last few days, I'm happy to report I should be in good enough condition to pull this off. If you're curious to learn more about my experience at OSCON and this presentation, please see my writeup on the LinkedIn Blog.

OSCON 2008 Wrapup

Matt Raible — Fri, 25 Jul 2008 10:05:08 -0600

This week, I attended OSCON 2008 in Portland, Oregon. I talked to someone who thought the conference had a very small Java presence. I noticed this too, but that's how it's always been. Interestingly enough, they also thought it had a small Ruby showing. I guess Perl, Python and PHP will always dominate OSCON. Of course, there's nothing wrong with that. I've always admired OSCON for the diversity of developers and languages.

Below is a list of my entries for all the sessions I attended.

If you attended OSCON, did you enjoy the show? What was your favorite session? I'd love to hear other's impressions of the conference and how it could be improved.

[OSCON 2008] The State of Lightning Talks

Matt Raible — Thu, 24 Jul 2008 13:34:08 -0600

From this session's detail page:

As a repeat of the last 2 years, 15 open source project leaders will be given five minute lightning talk slots to bring the audience up to date on their projects.

This year’s speakers and projects include: Brian Aker: Memcached, Glynn Foster: OpenSolaris, OSI: Danese Cooper, MySQL: Monty Widenius, PostgreSQL: Bruce Momjian, GNOME: Dave Neary, Gentoo: Donnie Berkholz, OpenOffice.org: Louis Suarez-Potts, Jabber: Peter Saint-Andre, Mozdev: Brian King, OpenID: Scott Kveton, Open Scrum: James Dixon, and Cliff Schmidt for the Talking Book Project. A couple more projects may be added later.

Each speaker has exactly 5 minutes, and we use various “fun” tools to make sure they stick to their time. The session usually ends with a “zinger” presentation worth staying for.

Unlike the description above, apparently there's 17 projects instead of 15. This session is 95 minutes long (spans 2 normal sessions). I figure this will be the most challenging event for me to blog, so here we go!

OSI (Open Source Initiative), Danese Cooper
Danese has been on the board of OSI since 2001 and will be on the board for another 3 years. This year, they've been asking themselves "What are the ways to make OSI more effective?" The best answer they've come up with is that they should run OSI more like an open source project (with lots of transparency). Minutes are posted publicly on website and they have a public blog. They're also transferring their archives to a searchable system. They'll be using Trac to do more objective tracking of all the issues that've been brought to their attention.

They're also going to establish (formal) membership. They're going to use Apache as a model and allow Charter, Individual, Organizational and have an Annual Members Meeting. ASF currently has 300 members and these folks are in charge of making decisions for the foundation. There's a good chance you'll recognize all the Charter Members of OSI. The names will be announced in the next couple months (I believe). There will be opportunities to be individual or organizational members.

There's new board members this year with an international focus: Nnenna (Africa), Harshad Gune ...honk... out of time.

Open Scrum, James Dixon
James is the founder and Chief Geek at Pentaho. Pentaho recently took their products open source and they're releasing Open Scrum as an agile methodology for managing open source projects. Open Source is not a methodology, it is a set of principles.

Some agile principles are used in open source, but some are not. For instance, trusting developers and face-to-face contact can be difficult with open source projects. Simplicity sometimes comes out of open source projects, but usually only for projects with a large developer base. There's a 45% good fit between Agile and Open Source.

Open Scrum is based on Scrum, but geared towards open source projects. It's simple and flexible and template driven. It scales from single-geek, part-time, single-projects to massively multi-geek full-on and multi-layered undertakings.

MySQL, Monty Widenius
Monty is the Chief Fonder of MySQL and he's talking about what's new in MySQL. There's been no new releases for users, but internally they've been doing lots of bug fixes in 5.1. He expects to have a reasonable GA of 5.1 in 3-5 months. 5.0 will include online backup and optimizer changes for faster joins and sub queries. 5.1 has an InnoDB Plugin.

MySQL was acquired by Sun for $1 billion and 6 months later, life is good. MySQL Backup was announced to be crippleware by MySQL management, but later changed to be open source.

Switched to use Bazaar and Launchpad to increase community participation. Started public bug hunt for MySQL 5.1. Started using case competition for new features in 5.1. There's also a new MySQL forge. Please help MySQL get better by filing bugs!

Maria engine 1.0 and 1.5; now crashsafe and multi-versioned. Transactional part to follow soon.

OpenID, Scott Kveton
Scott is on the board of the OpenID Foundation and his son is on stage with him (with a Twitter shirt on). In terms of adoption, they have a hockey stick graph with the big upturn starting in February 2007. OpenID is only 3 years old and there's over 20,000 sites that now support it. There's about ~500 million OpenIDs (including every AOL, Yahoo, MySpace and LiveJournal user).

OpenID 2.0 was released as final 12/7/2007. What's next: usability (currently sucks), security (PAPE), integration with other protocols, more than just providing, and possibly OpenID 3.0. One of the biggest problems with usability is people just can't remember URLs, so they're trying to figure out a URL -> e-mail mapping.

OpenID Foundation has added 5 corporate board seats and has 7 community board seats and elections are in August. You can get code at openidenabled.com and you can get involved at openid.net.

Gentoo, Donnie Berkholz
Donnie is one of 7 people that's on the council that tries to make sure Gentoo is going in a good direction. In the last year, they've been struggling with community and releasing. Gentoo is just getting old enough where people are quitting and new developers are coming on board. In 2005, they had a lot of voters for their council. 2006 and 2007 were bad years from a community participation standpoint. This year, they've fixed "poisonous people" and participating is back up.

Rather than voting for single candidates, they use a "rank all the candidates" voting system. Community is very important and they've recently kicked out 3 people that didn't think community was important. They released 2008.0 a few weeks ago. It was supposed to be released in February. When they released, they made the front page of Digg, which made Donnie very happy. Gentoo news announcements were way down in 2007, but now they're back up.

Recommended books for community building: Good to Great, The No Asshole Rule, Getting to Yes.

PostgreSQL, Bruce Momjian
Bruce is a PostgreSQL core developer and works for EnterpriseDB. The future for PostgreSQL is going to be a lot different than the past. PostgreSQL's path to the future isn't a straight line. Rather, it's a meandering path like a road up a mountain with switchbacks. People would like a catapult to put features into to get things in, but it's not that easy. It's difficult to improve software that's been around for 20 years.

Companies have very clearly defined goals that usually revolve around money. With open source, it's a lot different. Goals are usually for a feature set, for adoption of open source or for the challenge and fun of it.

PostgreSQL 8.0 was released in January 2005 and had a lot of enterprise features: Win32, Savepoints, Point-In-Time Recovery, Tablespaces. The latest release (8.3 in February) was much different and much more exotic features. In the future, they're going to expand on running queries on multiple CPUs and maintaining their leadership in the open source database space.

Bazaar, Mark Shuttleworth
Mark's interest in distributed version control predates his interest in Ubuntu. If we can elevate our process of version control, we can elevate the entire open source community. Bazaar is a distributed version control system that's developed in Python. It focuses relentlessly on a couple of key values:

Cross-platform: Linux, Windows, Mac OS X
Performance: sub-second status on > 10,000 files
Being adaptive to the way that people want to work (can use it like Subversion if you want or you can have distributed branches or have a team checkout)
Extensibility and embeddable - has a plugin system
Python Hackable - people can easily modify

Projects that use Bazaar: GNU, MySQL, Twisted and Ubuntu.

Memcached, Alan "Dormando"
Memcached is a very simple project with a mailing list with very simple people on it that run very simple websites. They haven't had a release in 6 months. Memcached was written by a guy named Fitz that wrote it and left. They've only recently created a developer community that's all over the globe. Alan didn't bring any slides, but he did release 1.2.6 RC1 as he was standing on stage. They also hope to release 1.3.1 this week, but there's not a lot of documentation.

People fork Memcached a LOT, mostly because they need a new backend. They've added a new storage engine that's pluggable to hopefully fix the forking problem. Also, their new binary protocol should make it a lot faster for high traffic web sites.

The Talking Bridge Project, Cliff Schmidt
While the network at OSCON has been slow, the folks that live in rural areas around the world have an incredibly slow means of getting information. Their network is often roads that people travel on to communicate with each other.

They've created a Talking Book device that is an MP3 player that costs $5-10. They're also creating kiosks that people in rural areas can interact with to download new information (i.e. podcasts and other educational content). It also has a microphone and can be found at literacybridge.org. They're creating open source projects to help build the device and content around it. They'd love to have open source developers join their projects and help them make access to information easier for rural societies.

Open Source Lab at Oregon State University, Lance Albertson
Lance is a Kansas native and grew up on a farm. He's a Gentoo developer and joined OSL a year ago. OSL was created in 2003 to utilize "extra" unused bandwidth. They provide managed and co-located hosting for open source software projects. They also provide on-the-job learning opportunities for students in system administration and development.

In addition to hosting, they have a development and outreach side. They have 12 student employees that are given developers/sys admins are given real world practical opportunities. Their work is seen and used by many people and students are given extraordinary amounts of responsibility. They're working on a couple external OSS projects: Oregon Virtual School District and One Laptop Per Child.

Outreach wise, they have an Open Source Education Lab and they've been doing Drupal training for OSU employees. They've also helping organize GOSCON, which is an open source conference for governments.

They currently offer hosting for over 70 OSS projects. Services include, email relays, databases, web hosting, file mirroring and code repositories. Notable clients: ASF, Drupal, kernel.org (master), OpenOffice, Freenode, Gentoo Linux & Debian, phpBB and many others. Datacenter currently has 50 racks; they'll be moving to 75 in the near future.

OpenOffice
They've had 150 million downloads and believe there's around 1 million users. There's 350 committers and most of them work from Sun. Presently, their biggest focus is Desktop. In the future, they'll focus on the Web. The idea is that people will want to use OpenOffice from their mobile devices. ODF is a format that can be expressed by any application. OO 3.0 is about extensions, not bloat. They're adding linkage to a free PIM (Outlook replacement). They're also adding better interoperatbility with other suites and formats (e.g., Microsoft Office 2007). They're adding productivity tools and adding toolkits to make things easier for developers. In the next year, they expect to go way beyond 150 million downloads. They also expect to expand greatly internationally.

Mozdev, Brian King
Brian has been in the Mozilla Community since 1999. Mozdev.org is a community site that provides free hosting for Mozilla applications and extensions since 2000. Established as a non-profit organization in 2004. It has over 250 active projects today and there's over 500 developers discussing issues on the project_owners@mozdev.org list.

Services provided: source code hosting, bug tracking (Bugzilla), communication tools (lists, newsgroups, blogs, forums), file management tools, project tagging, wiki, statistics, permissions system and public planning.

They've experienced some grown pains in the last few years, but they're mostly solved now. They've added developer resources to help develop their site, but they need more. Up until Firefox, they were the #1 place to go for Mozilla projects. They're still working on letting people know that they're around and have projects that are interested. They're looking for new avenues for funding.

Moving forward: they'll be add more items to roadmap, making it easier for less technical folks to host and create extensions, solving the funding issue, and developing deeper relationship with Mozilla (MoFo and MoCo). Quite simply, they want to continue with their goal of supporting developers in order to proliferate Mozilla technologies.

Glen Foster, OpenSolaris
Hey - I had a beer with this guy last night at Kells! Rather than focusing on the older Solaris users, they want to move to focus on getting kids to learn and use it. A couple of months ago, they released 200805 and reduced 5 CDs down to a single live CD. The packaging system is now network based. They changes the default shell to bash and changed to GNU tools as they default as well.

They introduced a new site - opensolaris.com. Since Sun released the source code 3 years ago, they've had a long evolution of getting the source out the door. They were using Teamware and doing much of the development behind the door. Early next month, they're moving to Mercurial and they'll be moving all the source outside the firewall. They'll be using Bugzilla as their bug tracking system.

Similar to Gentoo, they've had quite a few people in their community that don't like change. They're working on fixing this and have had good progress. Several years ago, they created a governance system before they had a community. This was a mistake and their new community-driven governance system should work much better.

GNOME, Dave Neary
GNOME is a desktop environment and set of utilities. It's the default on Ubuntu and Fedora. It's the face of Linux on the Desktop. It's also a platform for development. They have a vibrant ISV community that includes Adobe, VMWare and IBM. In addition to desktop, GNOME is also a mobile platform. They also act as a big tent that provides an infrastructure for projects that are part of the GNOME ecosystem.

The most important thing to Dave is the shared vision of universal access to anyone and everyone. Very important things to the GNOME project: usability, internationalization and localization, accessibility (Sun and IBM have done a lot of work). Accessibility is something they've really focused on in the last year. They've created an Outreach Project for Accessibility and have gotten funding from Mozilla, Canonical and Google. Mozilla has done a lot to fund them and help provide accessibility for Rich Internet Applications. To see if a website is accessible, try it in lynx.

Subversion, John Mark Walker
John is the Community Manager for openCollabNet and is not a Subversion developer. Subversion 1.4 was released in September 2006. There was much anticipating for 1.5 and was released last month. The people rejoiced.

What's new in 1.5? Merge tracking: easier to merge changes from trunk to branch and vise versa. You can also do sparse checkouts (replaces -N). Interactive conflict resolution so you can do it from the command line client. It also add changelist support. svn:externals is no longer required to be absolute (sweet!) and you can add peg revisions in your URLs. FSFS repos never change a rev after written to disk. Clients many now perform chained copy/move operations locally. It also supports moving multiple sources for copy and move. Client operations are now significantly more responsive to canceling operations. "resolve" subcommand replaces "resolved" (deprecated). Delete (remove) now takes a --keep-local option.

Now it's time for the fun stuff...

Ken Drachnik had to leave for an emergency and is unable to give his talk on GlassFish. An audience member (Carl Fogel) volunteers to do his presentation. It's very funny, especially since he's not a Java Developer and has never heard of GlassFish. The only thing I got out of this presentation (besides lots of laughs) is that v3 will be released in June 2009.

Drizzle, Brian Aker
Stored procedures, triggers, prepared statements and many other things have been discarded. "One CPU to rule them, and in the query cache, bind them." If you have a query cache turned on in our database, turn it off or delete it.

Drizzle is currently ~420K. The Master Plan is to rethink everything and not assume everything was bad. Also, they can reuse many other libraries w/o writing them from scratch. They don't have to ship every library because many are present on operating systems. In with the new: C99, Posix, package-lib. They're moving to a MicroKernal design and moving code to the edge. No new features will be in core. To add interfaces you have to remove code so there's an equal amount of code.

Multi-Core support: no new locks, remove old locks. Think today (burn memory, messages scales). They can leverage Sun because they can build machines with more cores than anyone else.

Field Types: Less primitives - now there's just one "blob". Removed the 3 byte int and add UUID/IPV4(6). They dumped ACL because we authenticate in clouds. KISS.

You can get involved with Drizzle at http://launchpad.net/drizzle (bzr branch lp:drizzle) and http://launchpad.net/drizzle-discuss (mailing list).

[OSCON 2008] Web Frameworks of the Future: Flex, GWT, Grails and Rails

Matt Raible — Wed, 23 Jul 2008 16:25:23 -0600

Below is the presentation I'm delivering at OSCON today. Unfortunately, I had to remove slides on GWT and Flex to fit w/in the 45 minute time limit. I hope to expand this presentation in the future, as well as continue to develop the side project I'm working on using these technologies.

view on slideshare | download

[OSCON 2008] Caching and Performance: Lessons from Facebook by Lucas Nealan

Matt Raible — Wed, 23 Jul 2008 16:18:15 -0600

Facebook is not just a Social Networking Site, they're a "Social Utility". They have the 4th most trafficked site in the world and over 90 million active users. Of those users, the average usage is 50 pages per day. There's currently over 24,000 platform applications. There's thousands of Apache web servers and hundreds of MySQL and Memcached servers.

The biggest problem with scaling for Facebook is the complexity. Connecting to all the databases is impossible. They have a very large codebase - their homepage has 500 library files and 10,000 functions. Scaling affects resources, particularly with regards to memory consumption and socket connection limits. Cache retrieval is ~10% cpu-user of most pages.

Caching Layers: $GLOBALS, APC, Memcached, Database, Browser Cache, Third Party CDN.

The Globals Cache is a PHP function called "cache_get". The Globals Cache works nicely in that it avoids calling APC and Memcached, but it still requires the overhead of a function call. APC (Alternative PHP Cache) is used for opcode caching (hundreds of included libraries, thousands of functions) and variable caching (hundreds of MB's of data). They use the APC for non-user specific data: network/school information, database information, useragent strings, hot application data, site variables and language strings (now the largest consumer of memory). They don't use it for User data because they don't send users back to the same server each time.

Friends page with a normal run takes 4050ms, with APC enabled it takes 135ms. You can also set apc.stat=0 to gain even more speed (128ms). To bust client-caching, they use APC+SVN with the SVN tag on the file and get the latest version from SVN and store it. Of course, this is a "prime the pump" thing that doesn't happen in production at runtime.

The next layer of caching is Memcached. Facebook currently utilizes over 400 instances of Memcached and has made contributions back to the project. They use Memcached for user-specific data: long profile, short profile, friends and applications. They don't use the timeout feature, but rather use cache invalidation on SQL insert and update. It's harder to do when writing your application, but it's easier to maintain in the long run. To make Memcached faster, they created a PHP extension that reduced PHP function calling overhead and allowed UDP support. The Memcached extension runs ~10% faster realtime than in PHP space.

Facebook likes for each page to render in under 250ms on the backend. To see how long a page took to load, you can mouseover the copyright at the bottom of the page, and a tooltip will show you the elapsed time.

This presentation is available online at http://sizzo.org/talks.

[OSCON 2008] Google XML Pages (GXP) by Harry Heymann and Laurence Gonsalves

Matt Raible — Wed, 23 Jul 2008 15:23:44 -0600

GXP is a templating system that Harry and Laurence developed at Google. It was original created by Laurence in late 2001 (Java run-time, compiler written in Python) as part of the AdWords rewrite. I'm attending this session because I heard from a Google employee that they were using WebWork + their proprietary templating framework for the view. My suspicion is that GXP is that framework. The presentation I'm listening to is available at the following URL:

http://docs.google.com/Present?docid=dcbpz3ck_8gphq8bdt

Google XML Pages has the following features:

static type checking
convenient parameter passing/modularization system
partial markup validation
automatic escaping of untrusted content
encourages functional style, discourages side-effects
internationalization support
lightweight runtime

GXP is an open source project as of today and is available at http://gxp.googlecode.com. It's used by AdWords, AdSense, Checkout, Blogger, Analytics, Reader and many more.

HelloWorld.gxp:

<gxp:template name='com.google.sample.HelloWorld'
              xmlns:gxp='http://google.com/2001/gxp'>
Hello, World!
</gxp:template>

Output:

Hello, World!

GXP has compile-time markup validation as well as static-type checking. GXP has native data types: for text/html, text/plain, text/css and text/javascript. It supports loops, conditionals, abbreviations, internationalization (i.e. <gxp:msg>Hello, World</gxp:msg>) with placeholders. You can call GXP in the same package using <call:GXPName>. To call something outside a package, you can use <gxp:import> to import packages or classes. You can also use a qualified XML namespace to access another package.

[OSCON 2008] An Introduction to Ruby Web Frameworks by Ryan Carmelo Briones

Matt Raible — Wed, 23 Jul 2008 12:33:08 -0600

Ryan is a Server Monkey / Code Sumari for Edgecase, LLC in Columbus, Ohio. A framework allows you to create re-usable code. Frameworks allow you to use encapsulation. Frameworks tend to be domain specific. For example, Rails works really for CRUD application, but not for others (i.e. Twitter).

Why Ruby?
Ruby has been Object Oriented since day 1. Ruby promotes Beautiful Code that's easy to read and maintain. Yes, MRI has performance issues. Matz has said "I'm a language designer" and has turned over the VM to others for Ruby 1.9. Another thing that might keep folks from using Ruby or its web frameworks is the libraries available. This is understandable, but it's being solved by alternative implementations. This includes YARV (the official 1.9 implementation), JRuby, IronRuby (not ready for production) and MagLev.

Rack
A framework that provides a minimal API for connecting web services and web frameworks. As a web application developer, this framework allows us to know about web services, but not worry about the details of talking to it. Below is a very simple Rack application.

class HelloWorld
  def call(env)
    [200, {"Content-Type" => "text/plain"}, ["Hello World!"]]
  end
end

Rack allows the handlers do the work and not worry about the web server abstraction. Handlers exist for WEBrick, LightSpeed, Mongrel, Fast-CGI and many others. As an application developer, it allows you to choose different architectures (threaded, evented, etc.). Ryan is talking about Rack first because it's used in all the other Ruby web frameworks.

Ruby on Rails
Rails is 4 years old now and was written by DHH when he was a contractor at 37Signals. Rails doesn't define and grand new ideas, everything has been done before (MVC, code generation, etc.). What Rails did is package everything in a unique way that makes it very easy to use. Rails has influenced a lot of what has come from web frameworks in the last few years. One of Rails' nicest feature is code generation. Ryan showed part of DHH's Create a weblog in 15 minutes video to demonstrate code generation. He noted that the minute he showed was picked because David said "Whoops!" and "Look at all the things I'm not doing". Rails popularized Convention over Configuration using naming conventions and load paths. While this is definitely a cool feature, I think most web frameworks have adopted CoC by now. Maybe not JSF, but who wants to use JSF w/o a framework on top of it anyway?

One warning about Rails: "The Golden Path" can get in your way. Rails is very Opinionated Software and that's how Rails works. As long as you follow that, you should be very productivity. If you decide to go off the Rails (i.e. namespaces), it can be difficult.

Rails uses a DSL in its models (i.e. has_many, has_one for relationships) and in the Rails router. It allows you to very simply map a URL to a controller/method. In addition to DSLs, Rails has first-class testing and its generators create stub tests for you.

Bad things about Rails: too much magic, moves to fast (too many releases).

Merb
Merb was originally developed by Ezra Zygmuntowicz to run alongside a Rails app to handle file uploads. It grew from there and became it's own beast. Merb is very much about using only what you need. It has "package repos" that allow you to add additional features. For example, merb-core doesn't contain an ORM framework, just a web framework. Merb also allows you to choose your ORM. It's standardized on Rack, so it can run on just about any web server. It also included "deferred actions" that allow you to send some URLs to evented web servers and others to threaded web servers. Merb eschews the "magic" that Rails has. It tries to stay away from making it's code a "monument to personal cleverness". Simple code scales better and runs faster.

One of the downers to Merb is that it's flexibility allows you to get down to the nitty gritty. However, it can be less productive than Rails because of its flexibility. Another downside is its documentation and examples are sparse. Merb is not recommended if you're just getting into Ruby.

Camping
Camping is a micro framework (< 4K) developed by why the lucky stiff. It's designed to develop small applications. You can do everything in one file and create prototypes very quickly. It uses Markaby to write HTML code in a builder-style fashion.

Since a Camping application is all in one file, it can be difficult to develop large applications. The solution is to write small apps and mount them in the same URL space. The only issue with small apps sharing the same space is they have to use the same database. One downside to Camping is there is no standard test framework. Mosquito was developed as a solution, but doesn't seem to be maintained.

Sinatra
Simple. Fast. Effective. It's designed to allow creating REST applications with minimal dependencies. Similar to Camping, it has one file for the entire application. Unlike Camping, Sinatra doesn't follow MVC conventions, so it may be difficult to port a Sinatra application to another framework.

[OSCON 2008] The Keynote

Matt Raible — Wed, 23 Jul 2008 11:29:21 -0600

This morning, I woke up awful early to polish my presentation, walked to the train station and rode Amtrak from Salem to Portland. The commute was great: there's nothing better than traveling with power and an EVDO card + the option to get a cup of coffee. After getting off the train, riding The Max and walking to the Oregon Convention Center, I'm now sitting in the Keynote at OSCON. Here's my notes from this session.

10 years ago, leaders of the free software movement got together and tried to figure out a way to help people understand how to get access to software freedom. As they talked, there was a gradual meeting of minds. Finally, one person suggested "Open Source". A few weeks later, there was a larger meeting of people and they heard about this term. It was an idea that changed the idea of software freedom and what free software was. We've come along way since then. Last year, we heard about open source and and it trying to find identity in corporations. This year, we're hearing about corporations trying to find their identity in open source.

The official tag for this conference is: oscon08.

Tim O'Reilly
While this is the 10th Anniversary of OSCON, it's also the 12th Anniversary of the O'Reilly Perl Conference (where it all started). Tim began his activism with Perl when it got on the web. He was thinking about the internet and the online world, from the beginning (when many others were coming from Linux). Open Source was almost named "SourceWare". Tim believes his biggest contribution is bringing Open Source and the Internet together.

"Keep your History" - make the things you put online accessible for years to come.

When OSCON first started, it was all about the OS Wars. Tim is showing a shirt with the famous Ghandi quote on it about "First they laugh at you..." and it has a Linux logo on the bottom. It's seems ironic that Microsoft is now one of the major sponsors of this conference (my thoughts, not Tim's).

Open Source Technology in the Enterprise. IT jobs are 2.3% of all jobs posted, according to the Bureau of Labor Statistics. Technology oriented companies (e.g., Google, Yahoo, Sun) make heavy use of Open Source (40% of all jobs posted by Y!). Open source is growing faster in non-tech companies. Of the open source technologies in the enterprise, the highest share of jobs is Linux (19%), followed by Perl, JavaScript and PHP. As far as the fastest growing, Django and Alfresco are at the top.

Three Big Challenges and Opportunities:

Cloud Computing
The (Open) Programmable Web
Open Mobile

Cloud Computing: Amazon Web Services, Google App Engine, The Engine Yard, etc.

Jesse Vincent: "Web 2.0 is Sharecropping"

Danny O'Brien: "If we want people to have the same degree of user autonomy as we’ve come to expect from the world, we may have to sit down and code alternatives to Google Docs, Twitter, and EC3 that can live with us oon the edge, not be run by third parties."

Basically, Tim is saying the that cloud computing is great, but it doesn't fit well with open source. This is primarily because if you build on a cloud, you have to be careful not to get locked into that platform.

Data is the "Intel Inside".

The Web is the Internet Operating System - the subsystems will be data subsystems.

Locking in data: iTunes and Amazon's Kindle. On the other hand you have Yahoo's BOSS, which is doing the opposite.

We Need the Open Web Platform! Tomorrow's Keynote, "Supporting the Open Web" will talk much more about this.

The Mobile Web has caused the "browser wars" to resurface. However, big companies like Google are putting a stake in the ground and saying "We believe in open". Net Neutrality and The Open Handset Alliance are two of Google's smartest strategic decisions. They understand how much they depend on the open ecosystem.

When we look at our success in the last 10 years, we can be really excited. But what's really impressive is how much we (as an open source community) is how we've risen to new challenges and challenged the openness of new platforms and industries.

Christine Peterson
Christine is the President of the the Foresight Institute. Christine was the person who suggested the term "open source" at the meeting referenced above. Unfortunately, my first battery died as Christine was coming on stage, so I missed writing down the first 10 minutes of her 15 minute talk. She's talking about the openness vs. privacy of keeping US citizens safe. She started her talk apologizing for the ethnocentricity of her talk and moved to quickly note that the e-voting controversy wouldn't have happened if open source software was used.

"Who would have guessed that the folks with the pocket protectors would turn out to be the ones with the right stuff?" -- LA Times

Founding Geeks: Thomas Jefferson (mechanical geek) and Thomas Edison (electricity geek).

You can't just complain about things. The fear is real. We can't just complain about how DC is solving problems, we have to step up and solve them ourselves.

"No Secret Software for Public Sensing Data."

Dirk Hohndel
Dirk is the Chief Linux and Open Source Technologist at Intel. He's talking about Moblin: Linux for Next Generation Mobile Internet. This sounds like something that has been talked about a million times before. Why is it interesting today? Because we're at an open source conference and open source is what makes it interesting.

When people look at Intel, they don't think of open source. However, Intel is very involved in open source and uses an open source methodology internally for their development process. They also have one of the largest grids powered by open source (~100K Linux servers).

Moblin is about the internet, about mobility, about flexibility and extensibility. What's happening today is the ideas of 10 years ago have become affordable to produce (for manufacturers) and purchase (for consumers). There's lots of proprietary ways to develop the mobile web, but it needs to be open in order to prevent lock-in (to a platform) and encourage innovation.

A year ago, Intel started Moblin. Initially, there wasn't a lot of interest from open source developers. The majority of interest came from companies, particularly hardware vendors. To Dirk, this was disappointing as he really wanted a community to guide the project and make choices about the platform. There's lots of Open Mobile/Linux efforts out there, but there aren't any that are truly open - with access to the source code and everything else you'd expect from an open source project. Intel was hoping to announce a cutting-edge infrastructure for Moblin here at OSCON, but they're a few weeks behind. They hope to be ready for soon.

"The hope that I have is the community takes this from us. Show us where to go. Show us where not to go. Help us get this right."

Tim O'Reilly interviews Monty Widenius and Brian Aker
Tim asks how it's going 6 months in. Monty responds that he's very happy they didn't have to go public and that Sun is still trying to figure out what they bought. One of the things difficulties they've seen about encouraging Sun's engineers to be involved in open source is some are hesitant about open sourcing their code. The biggest problem is engineers are afraid of the feedback/scrutiny that their code will receive.

MySQL was very unique as a company in that it was a virtual company, with most engineers working out of their homes. MySQL has become an enabling force for moving Sun to a similar model.

Monty is working on Maria (new storage engine) and Brian is working on Drizzle (a slimmer version of MySQL). Drizzle was inspired by a conversation when Brian was talking to Rackspace's CTO.

"Do less and then create extensibility mechanisms." -- Tim O'Reilly

OSCON: Where are the good parties at?

Matt Raible — Sun, 20 Jul 2008 12:28:39 -0600

It seemed to work pretty well for JavaOne, so let's try it for OSCON...

This week, the best Open Source Conference will take place in the beautiful city of Portland, Oregon. I went to high school just south of Portland (Salem) for a couple years and my parents still live there. Portland is one of my favorite cities, my parents are two of my favorite people and OSCON is one of my favorite conferences. Seems like an excellent combination for a really fun week.

I'll be flying to Portland with Jack and Abbie on Tuesday. Wednesday I'll be at the show speaking; Thursday I'll be there listening and likely blogging. I hope to catch a happy hour on at least one of those nights. Friday, my family and I will be heading to the Oregon Brew Fest followed by a night at The Kennedy School.

So when and where are the good parties at during OSCON 2008? Here's what I know about so far - I'll add to this list as comments start flowing in:

Wednesday: RedMonk at Kells (8-10), OpenSolaris/MySQL/Zend at DoubleTree Hotel (8-12)
Thursday: SourceForge Awards Party at Jupiter Hotel (6-9), BeerForge III at Bossanova (8-11)

As far as finding me at the show, it's probably easiest to follow me on Twitter.

Update: A reader e-mailed me about http://upcoming.yahoo.com/tag/oscon. This seems to be an excellent source of OSCON party information. I've updated the above list with the ones that look good on Wednesday and Thursday.

Apache 2 on OS X: Configuring mod_proxy and SSL

Matt Raible — Thu, 24 Apr 2008 10:58:03 -0600

I recently had to setup Apache as a front-end web server for multiple backend servlet containers. The backend containers serve up different web applications, and the Apache front-end unites them from a hostname and port standpoint. The following instructions describe how to configure Apache 2 on Mac OS X to proxy requests to Tomcat or Jetty running on localhost:8080. It also shows how to enable SSL on Apache and force it for certain URLs in your Java web application.

Apache comes pre-installed on OS X, so you should be able to start it by enabling "Web Sharing" in System Preferences > Sharing.

$APACHE_HOME on Leopard is /etc/apache2. On Tiger, it's /etc/httpd. If you've upgraded Tiger to Leopard, it's likely you'll have both directories so make sure you're modifying the right one. I lost a few hours figuring this out, so hopefully this knowledge will appease some googler in the future.

Configuring mod_proxy

Open $APACHE_HOME/httpd.conf and add the following on line 480 - at the very bottom, just before "Include /private/etc/apache2/other/*.conf".
```
#
# Proxy Server directives. 
#
<IfModule mod_proxy.c>
    ProxyRequests On
    ProxyPreserveHost On

    ProxyStatus On
    <Location /status>
        SetHandler server-status

        Order Deny,Allow
        Deny from all
        Allow from 127.0.0.1
    </Location>

    ProxyPass    /myapp    http://localhost:8080/myapp
</IfModule>
```
ProxyPreserveHost allows request.getServerName() and request.getServerPort() to work as if there is no proxy server in place. In other words, even though Tomcat is running on 8080, request.getServerPort() will return 80.
The most important line is the last one as this is the dictates the location of your applications. Add more lines as you need to add more applications.
If everything is configured correctly, you should be able to run sudo apachectl restart and navigate to http://localhost/status. If you receive a "forbidden" error, make sure your /etc/hosts has an entry mapping 127.0.0.1 to localhost (as one of the last entries), or change "Allow from 127.0.0.1" to "Allow from localhost". If you get a "Server not found" error, you can tail the error log at "/var/log/apache2/error_log".

One issue I've seen with mod_proxy is when a request comes in and the backend server is down. When this happens, Apache returns a 503 Service Temporarily Unavailable and it doesn't seem to go away after the backend server is restarted. It does resume proxying after a while, but I haven't determined what causes the proxy to come back to life. If you know a setting that forces mod_proxy to check for the backend server on every request, please let me know.

Configuring SSL

Open $APACHE_HOME/httpd.conf and uncomment the following on line 470:
```
Include /private/etc/apache2/extra/httpd-ssl.conf
```
Open $APACHE_HOME/extra/httpd-ssl.conf and change line 78 to:
```
ServerName localhost:443
```

In httpd-ssl.conf, change line 99 to:

SSLCertificateFile "/private/etc/apache2/ssl.key/server.crt"

In httpd-ssl.conf, change line 107 to:

SSLCertificateKeyFile "/private/etc/apache2/ssl.key/server.key"

In httpd-ssl.conf, add the following after SSLEngine on to allow proxying via HTTPS:
```
SSLProxyEngine on
```
Follow the Using mod_ssl on Mac OS X tutorial. For "Common Name/Server Name", use "localhost". You can download the source for mod_ssl (which you need at one point during the tutorial) at http://www.modssl.org/source/.
Run sudo apachectl restart and go to https://localhost. If you get a "Server not found" error, run sudo apachectl -t to verify the syntax of your config files or tail -f /var/log/apache2/error_log to verify there are no errors in the log files.

Forcing HTTPS for certain URLs
If you proxy requests from /myapp -> http://localhost:8080/myapp, request.isSecure() will return false. If you change it to /myapp -> https://localhost:8443/myapp, request.isSecure() will return true. I needed to figure out a way to have http://localhost/myapp go to http://localhost:8080/myapp and https://localhost/myapp to go http://localhost:8443/myapp. Even better, I wanted to configure things in a way so request.isSecure() returned the value based on the originally requested URL, not on the proxied URL. Configuration like the following would be ideal:

ProxyPass    http://*/myapp    http://*:8080/myapp
ProxyPass    https://*/myapp   https://*:8443/myapp

The solution I came up with is to standardize on secure URLs in my application. That is, use /secure/* as a prefix for all URLs that should be accessed via SSL. To follow this convention and force it, I added the following in my application's web.xml file:

<security-constraint>
  <web-resource-collection>
    <web-resource-name>Secure Area</web-resource-name>
    <url-pattern>/secure/*</url-pattern>
  </web-resource-collection>
  <user-data-constraint>
    <transport-guarantee>CONFIDENTIAL</transport-guarantee>
  </user-data-constraint>
</security-constraint>

Once this is in place, accessing http://localhost/myapp/secure/index.html will result in an error. Accessing it using https will succeed. Following this, you can change your ProxyPass rules to the following and all requests to /secure/* will be https; other requests will be sent to http. The order of the rules below is important.

ProxyPass    /myapp/secure   https://localhost:8443/myapp/secure
ProxyPass    /myapp          http://localhost:8080/myapp

If this isn't a good strategy for you, Tomcat has the ability to use a redirectPort (in server.xml) that auto-redirects from http to https when CONFIDENTIAL is used in web.xml. I'm not sure if this redirect will carry through values from a form post.

History Meme

Matt Raible — Wed, 16 Apr 2008 10:58:45 -0600

From my 17" MacBook Pro that sits at home and I don't use much on a day-to-day basis:

powers:~ mraible$ history | awk '{a[$2]++}END{for(i in a){print a[i] " " i}}' | sort -rn | head
148 mvn
71 cd
46 sudo
40 ls
27 java
23 ll
17 svn
16 echo
13 vi
10 mate

From my 15" MacBook Pro that I use on a daily basis:

mraible-mn:~ mraible$ history | awk '{a[$2]++}END{for(i in a){print a[i] " " i}}' | sort -rn | head
83 cd
81 ls
79 sudo
65 mvn
39 mate
28 svn
19 vi
15 rm
12 open
7 tail

Nothing too exciting.

Proposal accepted for OSCON 2008

Matt Raible — Mon, 17 Mar 2008 19:21:10 -0600

From an e-mail I received earlier this afternoon:

We are pleased to accept the following proposal for OSCON 2008.

* Web Frameworks of the Future: Flex, GWT, Grails and Rails

It has been scheduled for 16:30 on 23 Jul 2008.

My Abstract:

What if the choices in web framework was reduced to 4? If RIA are the way of the future, it's possible that these 4 frameworks are the best choices for this development paradigm. This session will explore these frameworks, as well as entertain many other's opinions on the future of web development.

RESTful backends are easy to create with both Rails and Grails. Ajax frontends are simple to create and maintain with GWT. Flex gives you flash and a pretty UI. If you're an HTML developer, Rails allows you to quickly develop MVC applications. If you're a Java Developer, GWT + Grails might be a match made in heaven. This session is designed to help you learn more about each framework and decide which combination is best for your project.

I'm really looking forward to learning about GWT and Flex in the coming months. If you have any experience (or opinions) about the abstract above, I'd love to hear it. The louder the better.

For those who haven't been, OSCON is one of those truly special conferences. Possible reasons:

It's only an hour from my parent's house
It's a beautiful time of year in Portland
It's always the same weekend as the Oregon Brewers Festival
It's a kickass conference with the greatest diversity of Open Source Committers

I'm going for all 4 reasons and even made a reservation to stay at The Kennedy School. Should be a fun show.

Selenium User Meetup Videos Posted

Matt Raible — Wed, 5 Mar 2008 21:19:23 -0700

Videos of last week's Selenium User Meetup have been posted on YouTube. You can watch them below or click on the video to watch it on YouTube. Enjoy!

Lightning Talks

Q and A Session

Last Night's Selenium Users Meetup at Google

Matt Raible — Tue, 26 Feb 2008 15:51:56 -0700

Last night, I attended the inaugural Selenium User Group meetup at Google's Campus in Mountain View. It was an excellent event, with many of the core committers on hand to present and answer questions. Each presenter had about 5 minutes to speak and we learned many things about the Selenium Project itself, what's coming in the future and how Google has standardized on Selenium as their integration testing tool of choice.

Patrick Lightbody started the meeting by going over a number of project statistics with pretty graphs and such. There were simply too many numbers to write down, so hopefully his slides will be published soon. I was pleased to see that Google did videotape the entire event, so it should be available online soon. I'll update this post when it it. Below are my notes from the event.

Jason Huggins, Test Engineer at Google
Selenium is a test automation framework. Some folks abuse it as a macro tool. There's two reasons Selenium became so popular: it was able to test Ajax before any other testing tool and it allows end-to-end workflow testing. Selenium works on any platform, with any browser and allows many, many languages. It's possible that other frameworks that are more focused are better. There are 4 Selenium products:

Selenium Core (TestRunner)
Selenium IDE (for Firefox)
Selenium Remote Control
Selenium Grid

Paul Hammant, Open Source Manager at Thoughtworks
Selenium 1.0 - beta this week. RC, Core, Grid and IDE together. 1.0 will be shipped in a few weeks. Compared to today, lots of bugs killed, documentation improved and a greatly improved Selenium IDE.

Selenium 1.1 will be shipped in a couple months. Selenium IDE will be enhanced to obey RC instruction ... becoming the best mode of operation when it ships.

Some experimental iPhone-Safari capability (dependent on SDK for iPhone).

Selenium 2.0 will hopefully be released this year. Roll in of WebDriver functionality and code. New model-based API. IE plugin and Safari plugin (not really a plugin, most likely uses AppleScript).

Philippe Hanrigou (http://ph7spot.com)
Testing is good! Selenium is awesome! However, end-to-end web testing is slow. It is and always will be slow. How do you solve this problem? You solve it the same way we've solved slow traffic in the past - by building more lanes. Rather than one browser testing everything, Selenium Grid allows multiple lanes and 20 browsers. Features include:

Faster Builds, Faster Feedback!
Easy Install and Everyday Use
No Need to Change Your Tests
Leverage Your Existing Computing Infrastructure

One of the best parts about Selenium Grid is you can download and have it running in 10 minutes or less. Selenium Grid comes out-of-the-box with Amazon EC2 support.

Jennifer Bevan, Lead from Selenium Farm Project at Google
As of Friday, Google has over 50 teams running over 51K tests per day on internal Selenium Farm. 96% of these tests are handled by Selenium RC and the Farm machines correctly. The other 4% are partly due to RC bugs, partly to test errors, but isolating the cause can be difficult. Selenium has been adopted as the primary technology for functional testing of web applications within Google. That's the good news.

The bad news is Google is pushing the limits of Selenium. Using Selenium (RC + Core) at this scale exposed certain issues not originally anticipated as high-impact. IE/XPath slowness has a high impaction given that can only run one test at a time. Tests can cause many conditions from which RC cannot easily or automatically recover (for example, tests that don't call stop() in every exit path). Unexpected browser dialogs, popups, etc. eventually cause timeout exceptions.

Googlers expect that RC will work for the most part, but they want it to be more reliable, with better performance. So they have:

Created utility methods to improve performance when examinging large tables, overlaying domain-specific languages, etc.
Deployed retry policies based on failure reason.

Looking to the future - Google has not yet reached their expected usage of Selenium RC. Some projects cannot use the Farm until RC supports session-level configuration (not server-level). Many just want RC to be more reliable. So Google will:

Continue to contribute to RC, Core and to user-created helper libraries.
Keep doing so until all failed tests are not Selenium's fault.

Dave Astels, Google (Driving Selenium with RSpec)
Using RSpec, you can create a very easy-to-read Story with Scenarios that can be read (and likely written) by practically anyone. Dave then uses a small script to load up the stories and run them in Selenium. When he runs the script, the scenario is spit out and test pass/fail information. Learn more at rspec.info.

Alex Chaffee, Mad Scientist at Pivotal Labs (The Selenium/Ruby Project that Must Not Be Named)
What is Polonium? It's also known as Selenium RC Fu or Selenium On Rails 2 or Funkytown. It has simple extensions to Selenium RC:

wait_for
element assertions
launching/managing servers locally

Blackbox testing you're sitting out of the box and send in stuff. In whitebox testing, you get to open up the box and look at stuff. With Selenium, you can do Graybox testing, where you are doing blackbox testing (against the UI) and querying your database (or other resources) at the same time.

Dan Faulich, Sr. QA Engineer at Redfin Corporation (How Not to Run a Successful Open Source Project)
The first thing you don't want to do is support everything. The second thing you don't want to do is write your project in JavaScript. While it's great that almost anyone can hack on it, running in a sandbox sucks. Don't use a language that's bound by other people's security restrictions. Don't roll your own multi-language remoting. It's written using XML + XSLT and its such a pain in the ass to maintain that Dan is the only one that fixes bugs.

Shinya Kasatani, Developer of Selenium IDE
Selenium IDE is a Firefox extension that can record and play back tests in your browser. It can translate the recorded tests to many languages.

Selenium IDE 1.0 adds support for TestSuites. Another new feature is better recording features - it detects when the DOM is modified. Shinya has a demo where he uses the new IDE to test the Dojo Dijit Theme Test Page. Apparently, this doesn't work in the current version.

The Goal of Selenium IDE is to get more people interested in test automation of web applications and to help their projects to be successful.

Haw-bin Chai, Developer at CommerceHub
XPath is a powerful selection took, but it'd be great if we could use something like "article 5" instead of the cryptic //table[5]/tbody/... syntax.

Why UI-Element? Traditional locations can be ugly and they don't convey purpose. UI-Element is a Selenium Core extension and has Selenium IDE integration. It's written in 100% JavaScript. Examples:

ui=frontPage::topStoriesCountry()
ui=listingPages::article(index=5)
ui=listingPages::articleSource(articleIndex=1)

The UI Map is in JavaScript shorthand. It contains logic to map ui locators to traditional locations. Each UI element implements getLocator(), then you add a page set and add an element - all in JSON Format.

Simon Stewart on Web Driver
What on earth is Web Driver? One of the main problems with Selenium is it's written in JavaScript and it's too easy to hack. WebDriver is an idealized browser which allows you to do browser things like call get(URL), findElement(By.xpath()) and getTitle(). WebDriver is similar to Selenium RC, but it's not written using JavaScript. It's written in the native languages for each browser, which allows you to break out of the JavaScript Jail. The IE Driver is written in COM (C++). The Firefox Driver is written as a Firefox extension. Safari uses Apple events.

Soon there won't be a WebDriver project ... because it will be part of Selenium!

WebDriver likely won't be part of the core until Selenium 2.0 later this year. One of the nice things about WebDriver is you can implement different browsers. Out-of-the-box, it will support all the major browsers, as well as HtmlUnit with JavaScript turned off.

I had a great time learning more about Selenium and how most of its problems will be solved in the near future. The beers afterwards weren't so bad either.

Update: Videos of this event have been posted.

Dynamic Language Shootout: Groovy vs. Jython vs. JRuby

Matt Raible — Wed, 20 Feb 2008 00:08:29 -0700

Travis Jensen has an interesting post titled Our Dynamic Language Shootout:

...
For a variety of deployment reasons, we've decided that whatever we choose will be deployed on the JVM. As a result, this comparison is for the JVM versions of the languages, e.g. JRuby, Jython, and, of course, Groovy, which has no other deployment option. I want to also clarify that I have the most experience with Python and I really like the language. There is no doubt that the language influenced me in my evaluation, but I really tried to remain objective in spite of that.
...
As I did the evaluation, I tried to come up with a broad spectrum of important information. Others at my company gave feedback on the important characteristics. In the end, these are the features that we felt were most important: the interaction between Java and the selected language, the IDE support, the learning curve, existing web frameworks, and the existing community support for the JVM implementation of the language.

His conclusion: Groovy.

I don't think it should surprise you at this point that we chose Groovy. Even being openly biases towards Python first and Ruby second (hey, it's cooler :), I could not, in good conscience, choose either of them for melding into our existing environment.

If I were starting from scratch on a project, my choice would be very different. If I wanted to target the JVM, I would choose JRuby (at least until Jython 2.5 and Django are available); if I wasn't targeting the JVM, then it would be, for my Python, but I'd be equally comfortable choosing Ruby.

Well written Travis - I look forward to reading more about the new life you're breathing into your stilted development practices.

Denver Tech Meetup, Consulting Panel and My Jobs Timeline

Matt Raible — Thu, 24 Jan 2008 13:39:38 -0700

If you live in Denver and are involved in open source (or simply technology in general), you should make it a point to attend tonight's Denver Tech Meetup. I'm planning on going for about an hour. From there, I'm heading over to the DeRailed User Group for a Consulting Panel at 8. If you're interested in moving from a full-time position to contracting, you should come. This is open to the public, so anyone can attend.

While I was thinking about things to talk about on the panel tonight, I started reflecting on the jobs I've had in my almost-11-year career in technology. Here's my timeline since college:

1997: Full-time at MCI Systemhouse
1998: Contractor for IBM Global Services (6 figures w/in 6 months of graduating!)
April 1999 - April 2001: Full-time for eDeploy.com (Friday lunches rocked)
May 2001 - October 2001: Contractor for Douglas County (introduced to Ant, Struts, etc.)
January 2002 - November 2002: Contractor for OnPoint Digital (100% remote)
December 2002 - August 2003: Contractor for Comcast Media Center
August 2003 - October 2003: Contractor for ResortQuest
November 2003 - June 2004: Contractor for Adams County
June 2004 - August 2004: Contractor for Open Logic
October 2004 - December 2004: Contractor for Oak Grove Systems
January 2005 - May 2005: Contractor for Xcel Energy
June 2005 - January 2007: Contractor for Virtuas
February 2007 - June 2007: Contractor for Checkerboard
July 2007 - Present: Contractor for LinkedIn

Phew - that's 14 jobs in 11-ish years! Notice that I've only ever had 2 full-time positions. So far, I have no regrets and really enjoy being a consultant. If you're interested in learning more about how I started Raible Designs or how I get contracts, you might want to read the following posts.

How I started Raible Designs (Nov 2002)
How do you become an independent consultant and get contracts? (Jan 2005)

If you live in Denver and want to learn more - show up at the The Hive at 8:00 tonight.

Update June 2008: My 3rd full-time gig started in May. Now I'm the Lead UI Architect at LinkedIn.

Denver Tech Meetup: January 24th at 6:30

Matt Raible — Fri, 11 Jan 2008 14:15:16 -0700

From the RedMonk Social Planner:

Why our holiday decorations are still up, I couldn’t tell you, but I do have the answer you’re all looking for: the date of the next Denver Tech Meetup. You remember, the Denver Tech Meetup? The User Group without the User Group? But with beers? Denver’s least ambitious monthly(ish) (ha!) gathering of technology workers?

Yes, I’ve failed spectacularly in my ambition to hold these things every other month, as was the expectation back in 2005, but I’ve been a little busy. And as I’m fond of telling our clients, a percentage of something is better than an entirety of nothing, right?

So ignore the inset picture - I posted it merely for nostalgic reasons; two Thursdays from today we’ll convene at our probationary new venue (my bartending friend is now way uptown) and frequent Jabber hangout, the Celtic. 18th and Blake, for the Denver folks in the audience. Still very easy walking distance for yours truly, not to worry.

Let’s you and me cram as many Tech Meetups in as we can before I flee back east before the hell that is the Denver summer. If things break as I hope they will next week and the week after, it’s possible that I’ll have my own not-so-captive audience to drag along.

See you at the Celtic. No Smithwick’s, sadly, but there is Murphy’s.

It's shaping up to be a good month for meetups: Silicon Valley next week, Denver the following. I'm participating in a Consulting Panel at the local Ruby on Rails user group that same night. Hopefully they won't mind if I have a couple cold ones beforehand.

Open Source Web Frameworks' Mailing List Traffic - June 2007

Matt Raible — Thu, 26 Jul 2007 14:12:29 -0600

Who knows if these stats mean anything, but it does make a pretty graph. Current mailing list traffic leaders in the web framework space: Rails, Flex and GWT. For those frameworks with dev and users lists, these stats are from the users lists. If you find these numbers to be inaccurate, please let me know.

Here's the numbers in case you want to create your own graphs:

Rails: 4056
Flex: 3558
GWT: 2305
Django: 1951
Wicket: 1718
Struts: 1689
Grails: 1307
MyFaces: 1283
Tapestry: 1268
TurbyGears: 797
Stripes: 206
OpenLaszlo: 189

Subversion Hosting

Matt Raible — Fri, 20 Apr 2007 08:24:13 -0600

In years past, I never had much of a need for source control outside of open source projects I worked on. Now, as I create more and more training materials and presentations - it's essential. While I could host a Subversion repository myself, it doesn't seem like it's worth the hassle. I'd prefer to have it hosted (and backed up regularly) outside of my house. This week I'm looking to setup svn.raibledesigns.com.

I'm not really looking to get my own Linux box hosted somewhere. I pay around $60/month to KGB Internet for raibledesigns.com, demo.appfuse.org and appfuse.org. To get my own "managed" box is somewhere around $300/month. When I say "managed", I mean Contegix-style where I can say "install this", "do that" and they handle all the sys-admin for you. So all I'm looking for is a reasonable SVN hosting provider that'll give me 1-2 GB for a reasonable price. What's reasonable? I'd say $25-50 per month.

I did some googling and there's a lot of Subversion hosting providers. I e-mailed a few of them with my main question - "can I point my subdomain at your servers?" A few of them have gotten back to me, but now I'm curious to hear from folks using these services. Are you using a Subversion hosting provider for your business? If so, which one?

I'm more interested in bad reviews than good ones - but if you're happy with a service, I'd love to hear about it.

The cheapest one I found is SVNRepository.com. However, it's been an hour since I e-mailed them and I haven't had a response. Nevertheless, $10/month for 5 GB, unlimited repositories, Trac instances, etc. sounds pretty nice.

Update August 2, 2007: I ended up going with SVNRepository.com (Level Two - $6.95/month) and I've been very happy with them. I'm using them for Raible Designs' artifacts (presentations, training materials) so I don't use it on a daily basis - more like monthly.

Upgrading to Ubuntu 7.0.4

Matt Raible — Fri, 20 Apr 2007 06:32:26 -0600

You have to love how easy they make this.

The time doesn't seem accurate as it originally told me it'd be about an hour. Regardless, I love how easy it is to upgrade Ubuntu from one version to the next. I wish openSUSE had a similar feature.

Update - 3.5 hours later: This might take a while...

Update - during FAC: It's failed twice now and left my OS in a corrupted state both times. I suspect the MADM (or whatever it's called) prompt at the end of the download. I've entered "all" and "none" and it's failed with both values. VMWare rocks - I'm so glad I didn't ruin a working system. I'll be sticking with 6.10 for a month or so.

... and Country Bry is right - calling it by it's code name vs. the version number is pretty cheesy.

OSCON: A beautiful time of year in Portland

Matt Raible — Thu, 20 Jul 2006 09:56:26 -0600

It's that time of year again for the excellent OSCON conference. Last year's show was great, but I spent far too much time in the conference, and not enough time enjoying summertime in Portland. I have a fondness for Oregon in the summer. I spent my last two years of high school in Salem (40 miles south of Portland) and remember loving life in August. This was likely due to the fact that it rained most of the rest of the year.

This year, I'm determined to enjoy Oregon more, and attend the conference less. I have a pass, and I'm doing a short 20-minute talk on "7 Simple Reasons to use AppFuse" (2:00 Wednesday in the Expo Hall). But that's about it. The rest of the week, I hope to enjoy myself and not be pummeled with any new technology or buzzwords. I might end up getting sucked into a session or two, but I'm hoping I don't. With any luck, I hope to visit both Edgefield and the Oregon Brewers Festival. Edgefield on Wednesday, Brewers Festival on Friday.

On Thursday, we're throwing another party at the Red Lion along with a host of other companies. Last year, we had a SourceBeat/Virtuas party at the Red Lion on the patio. The views of Portland were spectacular and it was definitely a good time. This year, it's a Geronimo Live! party. To register, please click on the image below.

Thanks to the many sponsors of this event - we appreciate your support.

Raible Road Trip #10 begins this Sunday. With any luck, I'll be able to snap some pics along the way and blog about our trip as we go.

[DeRailed] Denver Rails User Group Meeting

Matt Raible — Fri, 16 Jun 2006 07:37:20 -0600

Last night, Francis (Virtuas' Operations Manager) and I went to the local Rails User Group meeting, better known as "DeRailed". This group was started by a good friend of mine, Fernand Galiana, who obviously doesn't know how to blog. Since Virtuas is a sponsor, and I really like Rails technology, I felt like it'd be a fun meeting to attend. I wasn't disappointed.

The meeting was held 3 blocks from our offices at the downtown Tattered Cover. The first presentation was by Doug Fales, and he talked about his new soon-to-be-launched site called WalkingBoss. With this site, he integrates GPS Data (GPX files) and Flickr photosets to allow you to easily plot your trips on a map. It looks very easy to use, and somewhat inspired me to get a GPS device, and possibly a new (smaller) camera. Doug's blog has more in his most recent post. Unfortunately, I don't see that his blog has permalinks, so this might be hard to find in the coming weeks.

The 2nd talk was Ruby Meta Programming and was presented by Ara Howard. Since I don't know a whole lot about Ruby, this talk was over my head, but somewhat interesting nevertheless. There was about 20-25 people that showed up for this meeting and I found it to be a much younger and more down-to-earth group than the local DJUG. Furthermore, almost everyone went out for beers at the Rock Bottom afterwards, which is a lot different than DJUG (10-15 out of 75-100 usually go). Good meeting, good group of guys. Thanks Fernand!

Getting ready for the Sun Fire T2000 Server

Matt Raible — Sun, 9 Apr 2006 18:38:06 -0600

Like Jeff, Bill and Bruce, I signed up for the free 60 day trial of the Sun Fire T2000 Server. Why? Because I heard you can get one for free if you blog about it enough. My plans are to setup continous integration environments for AppFuse and Roller using CruiseControl. I also hope to do some performance tests b/w Java persistence and web frameworks. Finally, I'd like to some single-server vs. clustered server performance tests using Tomcat.

When we built our house way back in 2004, we had it wired with fiber. Even though we still aren't using the fiber, I also had them install ethernet throughout. Today, I finally took advantage of it. I moved my cable modem downstairs and hooked up RJ45 connectors on both ends, so I can now plug my office network into an ethernet outlet on the wall. Good thing my Dad is a network guy - he made it all pretty damn easy.

It was a great afternoon in Denver: 85 degrees, cold Fat Tires in the fridge, kids running around playing in the kiddie pool - and I got my house re-wired in under 30 minutes. The "old basement" will soon become a server room. Now I need a rack - for the T2000 as well as an old Dell Dimension Fedora box. Any suggestions?

RE: Marketing is the billion dollar question in open source

Matt Raible — Mon, 6 Mar 2006 23:14:20 -0700

Dana Blankenhorn has an interesting post titled Marketing is the billion dollar question in open source. I definitely agree with this. Good marketing of a project can make it successful, and bad marketing can kill it. It really is the hardest part of being an open source developer. Sure it's fun to work on this stuff until the wee hours of the morning, but if no one (including yourself) appreciates your project (due to your lack of marketing), it really worth it? Compare that to wild enthusiasm by your users and people writing articles about your project. There's a stark contrast there.

Some of the hottest open source projects are driven by marketing. Read more (and comment) on my Virtuas blog.

Denver Tech Meetup on March 9th

Matt Raible — Thu, 23 Feb 2006 20:15:47 -0700

Stephen O'Grady:

Ok, I know I dragged this out far too long - mostly due to travel concerns - but let's just pick a date and run with it. So the date for the next Denver Tech Meetup is now officially March 9th.
...
Venue will be - barring unforeseen circumstances - the same as last time, the Wazee Supper Club. It's easy to get to, close to some of the downtown offices, and most importantly, is my favorite.

If you don't know what the Tech Meetup is all about, I like to describe it as a User Group meeting without the User Group; there's no common affiliation other than we're all in tech (and even that rule can be bent , and no technical meetings - just the after-meeting beers/cocktails (and maybe food).

This meeting was a lot of fun last time. Matt Filios and I enjoyed talking to guys doing PHP, Rails and even some developers from the OpenSolaris project. I highly recommend attending.

OpenSuse 10.0 vs. Ubuntu 5.10

Matt Raible — Thu, 5 Jan 2006 19:22:57 -0700

Ever since I got a new HP Pavilion, I've been planning what's next for my Dell Dimension 8300. I decided it's probably best to retire my somewhat hosed Fedora Core 3 box (Dimension 8100) and replace it with a new Linux server. After talking with a good friend, I decided to go with OpenSuse 10.0 or Ubuntu 5.10. Steve was a good enough friend to burn me DVDs of both. Yesterday, I bought a new 160GB hard drive and last night I tried to install Ubuntu. I went w/ Ubuntu b/c Steve tried them both and said he liked Ubuntu a lot better. I've never used Ubuntu, and I have used Suse a fair bit - so I figured I'd try something new.

When I started installing Ubuntu last night, I figured it'd be a breeze. I have a DVI KVM Switch hooked up to a Logitech cordless keyboard/mouse, and Ubuntu immediately recognized them both. However, at 44%, it failed to install gstreamer0.8-jpeg and the installation bailed out. I was able to login to the desktop and (seemingly) get stuff working, but I'm always a bit leary about a failure in the middle of an OS install. After an hour of futzing with it, I tried again and got the same error. Around 1 a.m., I said "screw this" and threw in the Suse DVD.

I had the same good results with Suse, where my keyboard and mouse were recognized. However, when I got prompted for the root password, my keyboard quit working and I was up shit creek. I started the re-install process before going to bed at 2 and picked it up again this afternoon - after a beautiful day of skiing at Copper. I got almost everything working on Suse this afternoon, and just as I was about to call things good - the keyboard problem came back. Pretty disappointing since I'd just gotten my Apple Cinema Display to work.

As I speak, I'm trying Ubuntu again, without the KVM switch. I suspect there's probably a piece of hardware I have that's causing the failure, so hopefully unplugging things will solve the problem. If I don't get it figured out in the next hour or two, I'll probably just go with Suse, setup VNC - and get a wired keyboard for when I need direct access.

24 hours later: It's interesting to see that almost the commentors on this post are recommending Ubuntu. After posting this, and receiving a comment from Brett, I tried the Ubuntu Live DVD. What I found was that Ubuntu recognized my cinema display, but it entered into a non-stop flickering loop that I couldn't solve. Therefore, I threw in the Suse DVD and tried again. This time, Suse recognized everything flawlessly (including my HP OfficeJet G85). So I'm sticking with Suse - mainly because it seems to recognize my cinema display, printer and DVI KVM switch the best. With apt-get working on Suse, it's been a breeze to get everything setup.

January Denver Tech Meetup

Matt Raible — Tue, 3 Jan 2006 15:10:22 -0700

Hey Stephen, when's the next Denver Tech Meetup? I propose next Thursday the 12th. Whaddya think?

Open Source CMS Evaluation - Part III: Implementation

Matt Raible — Wed, 28 Sep 2005 12:50:16 -0600

In my last post, I narrowed my open source CMS candidates down to Joomla and Drupal. I was hoping to have a choice made by Monday morning, implement the design in the morning, and populate the content in the afternoon. Two days later and I'm now where I was hoping to be on Monday morning. I've spent the last two days implementing both Joomla and Drupal. Monday, I spent most of the day with Joomla. While it was easy to apply my own theme, I became very discouraged when I discovered I didn't have full control over the HTML markup produced. All of the content I produced was wrapped with a <table> - and from what I could tell, it was impossible for me to change that w/o hacking Joomla's code.

Based on that discovery, as well as the overwhelming number of pro-Drupal comments I received, I moved on to implementing Drupal. Monday night and yesterday were spent with Drupal. It's been extremely frustrating, but mostly because of all the CSS I had to write. The major problem with Drupal is the admin interface uses the same template as the reader interface. I did find a nice way to use an existing theme for the admin, and our own for the reader - but decided not to use it because it would give content authors the wrong impression of what their stuff looks like.

The majority of the time I've spent with Drupal has been modifying templates and installing modules. For the most part, Drupal doesn't come with everything you might need. I found the CivicSpace download to be much more complete with modules I needed. In addition, it has an installer which makes things a bit easier to setup for a web designer. I'm currently using the Article module, which works quite well, but I wish I could create multiple blocks for different categories (taxomies). Instead, I had to hack up my own block using some SQL to select all the "news" content types (for a Recent News block).

My biggest problem with Drupal continues to be my lack of knowledge. Luckily, there's a plethora of information out there and a lot of people are using it. I've been able to use the Drupal Forums as well as Google to solve most of my issues. Now the hard part comes - I need to show it to the designer/marketing folks and teach them how to use it.

The brochure site in an hour tutorial was extremely helpful for me to get started with an About page, Contact Us page, and Press Releases. However, it says to use "books" to create pages, and I've seen others recommend "page" and "story". So which is the best one to use? Should I advocate using "page" for regular site pages, and then "story" for our articles and whitepapers? Or should we use "book page" for the main pages. I'd like to limit the number of choices if possible.

I think the major problem with using Drupal is going to be tweaking our template. Every time I see a new custom theme (like this one) I want to steal stuff. Right now, I'm using a design from oswd.org and much of the CSS from the spreadfirefox theme.

Conclusion: No CMS is perfect. You'll have to hack it on one way or another to make it fit your needs. Drupal seems to be used by many web designers w/ little to no programming skills. Most folks love it and I've received many, many positive comments about it. I've received hardly any positive comments about Joomla. Zope and Plone also seemed to inspire hatred among some users.

Lesson Learned: Listen to your readers. Other users' experience is one of the most valuable indicators of a good open source project.

Open Source CMS Evaluation - Part II: Customization

Matt Raible — Mon, 26 Sep 2005 01:32:58 -0600

This is the third and final post in my quest to find the best open source CMS for my needs. Previous posts include Building a website with an Open Source CMS and Open Source CMS Evaluation - Part I: Installation. Based on these two posts, reader feedback, and my installation experience, the final round of candidates include Joomla, Drupal, Magnolia, OpenCms and MeshCMS. These are listed in the order that I expected the final rankings to be - just to let you know what my feelings were going into this final process.

The CMS I choose will be used to build Virtuas's website, as well as customers of Virtuas. By using a CMS to produce websites, the design is separated from the content - and the "adding content" process can be much easier for the customer. This greatly simplifies the "creating a website" process for us, and will likely save our customers a fair amount of design costs. In addition, it makes it much easier for site owner's to maintain the site after it's been published.

As for Virtuas's site, it's static right now, and we'd like to change that. We want to ability to show recent blog posts on the front page, as well as make it easy for Practice Leaders to publish articles. In addition, it should be easy for our designers to change the design (1-2 files) and for our marketing team to add press releases and update existing content. Tomorrow, I'll be presenting my choice to the rest of the team, and we hope to design and start publishing content this week. Our goal is to have a new Virtuas site up and running one week from today.

My goal today was to see how easy each CMS was to customize. In addition, I wanted to see how easy it was to publish an article, as well as to aggregate our latest RSS feed titles onto the homepage. To test the design customizability, I tried to reproduce the current Virtuas homepage. Then I published Jeff's Geronimo Article, and attempted to aggregate feeds from Maria's and Bruce's blogs. My main reason for putting the Java CMS'es lower than the PHP ones in my suspected order of finishing is because I don't they don't have the RSS Aggregation feature.

Rather than just jumping in and using each CMS in anger, I tried to start off by reading the documentation for each. My main focus was on how to customize, but I also looked for an RSS Aggregation feature and ease-of-publishing for articles. I read documentation for 15-20 minutes, then dived into creating a custom theme and adding content. I installed each CMS on my PowerBook, and used Safari and Firefox on OS X, as well as Firefox on Windows XP in some cases.

MeshCMS - I spent 40 minutes looking into MeshCMS before I knew it wasn't the one. The main problem I had with it was the upgrade process. To upgrade to a new version, they recommend that you use symlinks to your files and store them in a separate location on the file system. While this may work for some, it seems a little brittle to me. I'd rather use a solution that keeps everything stored outside of the application by default. Creating a new theme was quite easy - but there didn't seem to be any support for multiple menus (i.e. global and local navigation), nor was there any means to customize the menu template.

I did manage to blow up the whole application at one point, simply because I was missing a JSP tag in my template. Since I had this template selected for the administration as well - it hosed the whole application and spit out stack traces for each page. Luckily, renaming the template directory caused MeshCMS change to revert to the default settings and everything was fixed. The interesting thing about MeshCMS is it looks very similar to the SiteMesh+JSPWiki CMS I wrote a few weeks ago. However, mine allowed full menu creation by editing/creating wiki pages.

OpenCms - When I installed OpenCms a couple of days ago, I initially did it on Windows. Everything worked fine and after waiting 18 minutes for everything to import, I was able to browse and edit the default site. However, today was a different story. The version I installed on my Windows XP box no longer works. When I got to http://localhost:8080/opencms/, I get a directory listing with "resources" and "setup" on it. The same day I installed OpenCms on Windows, I installed it on my PowerBook. It took 38 minutes to complete, but nevertheless, it said the process worked. Today I re-ran the setup and now I have the same result as on my Windows XP box. If the setup and installation is this fragile, I'm not interested. Blame me and the fact that I'm a redneck all you like, but the Magnolia installation is still functioning just fine.

Magnolia - I spent about a half hour with Magnolia before I knew it wasn't for me. While the admin UI is impressive with all it's Ajax goodies, creating a new template is cumbersome and not designer-friendly at all. You have to create a "new node" and then a bunch of "data nodes" under that. The documentation (a QuickStart PDF) is 17 pages long and forgets to mention the "title" data node is needed before the template will show up properly as an option. Once you've created a new template in the admin UI, you have to create the template on your file system - inside the web application. This may make it difficult to upgrade if you're deploying Magnolia as a WAR. The worst part is after creating the template, you have to restart the server. WTF? That seems a bit ridiculous to me. Granted you'll likely be designing your master template in a development environment - but good luck installing Magnolia for a client and having them create a new template.

One of the most interesting things about Magnolia is most of the folks who've recommended it have highlighted that it's "built on the revolutionary Java Content Repository Standard JSR-170". While I can admire the technical merits of this effort, it doesn't necessarily make this a good product. A good product, IMHO, is easy and intuitive to use. The admin interface for Magnolia is not intuitive. I like the fact that I can right-click on a page/node/etc., but on my Mac (with Firefox and Safari), the real context menu shows up on top of the application menu after a second or two. I'm going to pass on Magnolia due to the fact that its not designer friendly, as well as the fact that templates can't be edited in the browser. It looks like something that might be very interesting for developers, but it's simply not friendly for HTML developers.

At this point, it's 10:30 p.m. on Sunday night. I need to make a decision before I go to bed tonight and I'm scheduled to meet with our designer at 7:00 a.m. to start implementing his new design. I haven't started the PHP options, and I've had a couple new ones recommended on my blog while doing this evaluation today. Mal recommended Exponent and Jacob recommended MySource Matrix.

Because I'm down to two choices (and I haven't tried to customize either one), I decided it was worth looking at both of these PHP solutions. Exponent installed easy enough, but MySource Matrix failed miserably. Joomla, Drupal and Exponent all had an easy-to-use web installer that *just worked*. MySource spit out a bunch of permissions errors (even after chmod -R 777 *) and told me I had to run .php files from the command line. Since the other options all installed easily, I decided not to continue evaluating MySource Matrix.

Exponent - I didn't spend very long looking at Exponent. At first, I didn't think it had any documentation b/c it was a bit difficult to find on their site. Maybe it's because they don't have a background set on their site and my browser defaults to gray - making the gray text difficult to read. Even w/o documentation, I was able to navigate around the default Exponent site and figure out how to edit content. It's an easy UI to use, but again I was disappointed to find the "corporate" theme doesn't have a white background. Most good web designers know to set a default background color - and it always annoys me when someone misses this step. It's possible they don't set the background on purpose - like Yahoo does.

The deal-breaker with this CMS was that I couldn't edit any files w/in my browser to change anything (all I wanted was a white background). While it's theme management and templating looks powerful - it's another file-based system where you have to configure everything and then upload it. I agree that this is likely the path that web designers will want to use to get started - but I think it's important that files can be tweaked on-the-fly. Using a good FTP tool is certainly an option, but I'd prefer theme-editing to be part of my CMS. The one thing I did like about this CMS was the clean URLs. Granted, they aren't static-looking by any means, but having a simple ?section=# seems cleaner than the multiple parameters that other systems use.

Drupal - This CMS seems to have a lot of things I want/need as first class citizens. A blog, news feed aggregation and the ability to provide pretty URLs (aliases) for more cryptic CMS-type URLs. I couldn't get the URL aliases to work, but I suspect I was doing something wrong and didn't give it enough attention. I didn't spend a whole lot of time with this CMS, but rather just browsed around the admin interface and read a bit about how to create themes. I installed the PHPTemplate engine, but never installed any themes. When I found out I couldn't edit any of the uploaded templates contents, I started to get discouraged by Drupal. One thing I found disappointing, with both Drupal and Joomla, is they seemed to hard-code my server name into many of their URLs. When I installed these applications on my PowerBook, I used "localhost" for the server name. When testing out things from my Windows box, I couldn't even login to Drupal b/c it kept redirecting me to "localhost". Joomla had a similar problem with localhost, except that it only screwed up stylesheet paths. I was still able to administer the application.

Joomla - My time with Drupal was short-lived, mainly because I was itching to start playing with Joomla - which I've heard a lot of good things about in the past week. Furthermore, it's got a really good-looking administration UI. It's the type of UI that a designer would look at and appreciate. When trying to edit pages from my Windows box - everything worked, but I couldn't save the page. No JavaScript errors or anything, there was simply no reaction. Editing pages and content from my PowerBook solved the problem. I was able to easily create a simple theme that looked like virtuas.com and upload it. The theme isn't perfect, but it was easy enough to create using the Velvet theme from Joomlashack.com as a template. The weird thing about Joomla, at least with the default install, is there's no notion of pages. Everything is some sort of news item. In the pages I created, I was also unable to remove all the authoring notation and other junk that I don't want to show. The admin UI had options to remove the stuff, but even after "applying" the changes, they still showed up in the reader view.

It's now 1:30 a.m. on Monday morning and the last three CMS's definitely didn't get the attention they deserved. Nevertheless, I think they're the best of the bunch. Not only were they much easier (and quicker) to install than the Java options, but their UIs are also good-looking and easy to use. Drupal and Joomla both seem like excellent choices. Drupal seems to be more of what I'm looking for since it has all the features I want, and allows aliasing of URLs to make it appear like a static site. However, Joomla is a lot more eye-catching and that alone makes me want to use it. Neither of these CMS'es seem to have a full-featured blogging engine, at least not one that's as good as Roller.

Conclusion: I'm going to recommend we use Joomla, and look into doing some URL Rewriting to pretty up it's URLs. I doubt there's a whole lot we can do, but I'd like to figure out a way to make them a bit more search engine friendly. Drupal seems like an excellent choice as well, but the fact that I can't edit templates from the UI kinda sucks.

Thanks for listening y'all - all your comments and feedback during this evaluation have been great.

Open Source CMS Evaluation - Part I: Installation

Matt Raible — Fri, 23 Sep 2005 17:39:04 -0600

Today I began my journey in evaluating Open Source CMS applications. The motivation for this adventure can be found in my post titled "Building a website with an Open Source CMS". Basically, I'd like to find a good solution to build small-to-medium size websites. I want full-control over the design and features, and it shouldn't be too hard to configure, install or administer. I received many suggestions on my initial post, and thanks to these comments, I'm considering the following CMS solutions:

Alfresco (Java)
Daisy (Java)
Magnolia (Java)
Bricolage (Perl)
MeshCMS (Java)
AtLeap (Java)
Mambo/Joomla (PHP)
Drupal (PHP)

The reason OpenCms didn't make this list is because of Tim Howland's first demo (he left the link in a comment). As soon as he dropped into a shell, I gave up. I also tried OpenCMS a couple of years ago and couldn't get it to work. I still have a bad taste in my mouth from that experience.

Of this list, I was able to eliminate 3 options quite quickly: Bricolage, MeshCMS and AtLeap. Bricolage b/c their site was down for days when I first started looking at CMSs. In addition, it's written in Perl - of which I've never written a single line - and I don't feel like I'd be well suited to customize a Perl product. As for MeshCMS and AtLeap, both of these were eliminated b/c of the lack of mailing list traffic. This represents a small community IMO and I'd hate to start using a product that's not well supported or well documented. This is unfortunate b/c I could probably customize these products easier than the rest. Nevertheless, it's my customers that are important, not me.

This leaves me with three Java solutions (Alfresco, Daisy, Magnolia) and two PHP solutions (Joomla, Drupal). I'm not too keen on including Alfresco b/c I've never heard of it, but readers of my last post recommended it, as well as my colleague Bill Dudney (another friend who forgot how to blog). At first glance, there are a couple of major differences between the Java and PHP solutions. The Java ones definitely take the cake on download size:

Daisy 1.3.1: 59 MB
Magnolia 2.1: 13.4 MB
Alfresco 0.6 (with Tomcat): 30.3 MB
Joomla 1.0.1: 1.7 MB
Drupal 4.6.3: 447 KB

Yes, large downloads is the nature of Java applications - but it's interesting to see the wide discrepancy b/c the 3 Java options listed here. In addition, it's a little disappointing that I can't download Alfresco standalone - why does it *have* to come with Tomcat or JBoss? Looking through the Tomcat directories and files - there doesn't seem to be any Alfresco-specific settings or files. Why can't I just download the WAR?

My goal in this session was to install these different CMSs and see which were the easiest to install - as well as which had the features I need to continue evaluated them. What I found was that Daisy is out of the running. This is primarily because it's a wiki, not a CMS. No wonder the JSPWiki guy was impressed with Daisy a few weeks back. Furthermore, the number of things you have to do to get Daisy installed is very long.

Magnolia, on the other hand, was very easy to install. Drop two WAR files into Tomcat and 10 minutes later you have a CMS. Yeah, that's a little scary - it took 10 minutes (647740 ms to be exact) to startup Tomcat after installing Magnolia. While I appreciate the easy installation, I'm still interested to see what kind of data store Magnolia is using. Is it using an embedded database or what? I didn't have time to look, but rather played around with the admin console a bit. I was very impressed with the admin console - even though I didn't figure out how to edit the homepage. It appears to be using Ajax everywhere. Right-clicking to edit a page even brings up an application menu (rather than a browser menu). Magnolia is impressive at first glance and will be included in Part II of my evaluation.

Alfresco, with a 0.6 release, was almost off my radar. The version number makes it seem like a very immature product - and the fact that it requires JDK 5 makes it seem even more immature. Upon startup, it spit out a number of errors - but they seem to be well documented in its "README_tomcat_linux.txt" file. I used OS X and despite the errors - everything appeared to be working OK. However, when I pulled up http://localhost:8080/alfresco, I immediately understood why Bill likes it. It looks like it's using JSF, and I'm willing to bet it's using MyFaces (he's a committer on the project). The disappointing thing I noticed after pulling up the initial page is that it's a login page. I'd expect to see a reader view initially rather than an admin view. After logging in, the interface seems very nice and easy to work with. However, after 30 seconds of clicking on stuff, I can't figure out how to get the reader view to show up - so I give up. I'm going to let this CMS graduate to Part II, but only because I didn't spend much time with it - and also because the UI looks quite polished.

I installed and played around with Daisy, Magnolia and Alfresco all in a 3-hour period today. First of all, I'd like to give props to all the authors of these OS projects as each was installable according to its instructions and I didn't have to google for a single setting. I installed Joomla and Drupal last week - both in under 10 minutes. Joomla was definitely more impressive - mainly because it's default homepage (note: this can change every hour) looks pretty cool. Drupal, on the other hand, is a bit more plain jane. This is not a bad thing necessarily - as it might be easier to design with a clean slate rather than remove-features, then-design. I did have to install PHP4 on my Mac to run both of these packages, but Server Logistics made that super easy. I also tried to install Joomla 1.0 on my Windows box (which has Apache 4.0.7 and PHP 4.3.3 from BlueGlue), but it failed halfway through the install with errors that my user/pass for the database were wrong. Not a big deal, but frustrating since it works fine on the Mac (and yes, the user/pass I'm using are correct).

Watch this space. Part II will address customizing each CMS to use an already-created theme, as well as playing around with each's features. I'd prefer a package that has built-in support for multi-author blogs - so that might make Magnolia and Alfresco look bad. However, I'm not going to hold it against them since I'm a Roller fan and committer. I don't mind using a 2nd application for blogging - especially if the built-in package is less than full-featured.

Conclusion: Alfresco, Magnolia, Joomla and Drupal are the easiest open source CMS applications to install (of the ones I looked at). Daisy was easy to install according to its instructions (of which there were many), but it's more of a Wiki than a CMS.

Update on Saturday at 1:00 p.m. - Based on reader's feedback to this post, I went ahead and installed three more: Plone, eZ publish (gotta love the domain name) and OpenCMS. All the previous CMS applications I installed on OS X, and I installed these 3 on Windows XP. I installed OpenCMS first, and while the setup was easy, it took 18 minutes to finish its importing of data. My Windows machine is easily 2x as fast as my PowerBook, so my guess is this would've been quite painful on the ol' Mac. Regardless, it's a one time wait-fee, so I can't really ding them for that. I played around with the Admin UI for about 5 minutes and found it extremely easy to use - as well as intuitive. I dig how you can view the content and edit it very easily. I never had to read any documentation to figure out how to work this system.

Next up was eZ publish. It took a bit of fiddling to get the installation to work. I had PHP 4.3.3 installed with Apache 2.0.47, and I upgraded to PHP 4.4 before trying anything. While setting up eZ publish, it told me it didn't work with 4.4, so I backed down to 4.3.3, after which it told me I needed at least 4.3.4. Thankfully, PHP is easy to install and upgrade on Windows and I got it all working fairly quickly. Of the three (and all the CMS's in fact), eZ publish has the nicest installer. The whole application and process gave me the feeling that it was the product I was looking for. It's designed to help you setup a website, and allows you to choose templates, features, etc. along the way. At the end, it stalls for a couple minutes - probably b/c it's creating everything and populating the database. The first thing I didn't like was the URLs (/index.php/feature). I wonder if that can be changed to be more path-based? Having an extension followed by additional slashes just doesn't seem right to me. From there, I logged in as an administrator and started playing around with things a bit. The Weblog feature seems very inadequate and doesn't even seem to support RSS. In addition, I couldn't figure out how to edit any of the templates in my browser. But the worst part was the admin UI was sssllloooowwww. It could have been b/c I had other stuff running, but I don't think so. eZ publish seems like a good system to recommend to friends who want quick and easy websites, but I don't think it's one for me.

The last CMS I installed was Plone. It was easy to install on Windows, but only b/c it has an all-in-one installer. I found it a bit strange that it doesn't install in a web server, but rather contains everything as part of the package. This includes Python, its own web server, and its own database (I'm assuming). It's an easy install, but I like the idea of installing a CMS into an existing web server rather than having a CMS with an embedded web server. The admin UI is simple enough to use, but it's very boxy and seems like it would be difficult to customize into a corporate or small business website. Please let me know if you feel this to be untrue (pointing to existing nice-looking installations would help). I didn't like the fact that it highlights members quite prominently, but that can probably be fixed with some good template-tweaking. The most disappointing thing I found was that I was unable to edit "skins" within the admin console. You have to download a separate package, customize it yourself, and then install it. What a pain, especially since I'd really like to be able to 1) modify template, 2) save and 3) view - just like this site allows me to do. For this reason, as well as the embedded server feature, I'm going to have to pass on Plone as well.

Conclusion 2: Alfresco, Magnolia, Joomla, Drupal and OpenCMS are the CMS applications I will continue to evaluate this weekend. I have to make a decision by Monday morning so I can start building a site with it. If any of these CMS'es don't allow me to customize its templates from a browser, please let me know so I can take it off my list.

Update on Sunday at 9:30 a.m. - I'm going to drop Alfresco and add MeshCMS for reasons stated here.

Building a website with an Open Source CMS

Matt Raible — Mon, 12 Sep 2005 11:48:14 -0600

I think the open source community has done an excellent job of figuring out how to create frameworks for developing web applications. But what about websites. You know, the web presence that every company wants - for minimal cost. For most companies, it's nothing more than 5-10 pages that tells a bit about the company, show some management folks and tells you how to get to their offices.

I've developed many websites over the years, many that were static, but more that were dynamic. The unfortunate thing about all of them is they required someone technical to update them. Often, the client had to contact me if they wanted anything new on the site. I've often thought there was a better solution - and I think I'm at a point where I know what customers want, and I know how to provide it. The solution is a Content Management Solution (CMS). One of the biggest problems with static websites is they're not dynamic enough. A CMS can alleviate this problem by reducing the bottleneck that a traditional "webmaster" creates.

In my mind, there are a couple of things that make a good CMS: 1) it's open source (to minimize costs to the client) and 2) it's easy to customize. On the customization front, my demands are a bit more rigorous - mainly because I know what many folks want in a website. Here are my main criteria for a good open source CMS - when it's used to power a regular ol' client-updateable website:

Design customization: I should be able to customize all the (X)HTML that's generated using one or two files (like SiteMesh allows). It should be possible to change the look and feel of *everything* by modifying some CSS. It should also be possible to use Mike Stenhouse's CSS Framework to simplify layout choices for clients. Ideally, a web designer or regular ol' HTML person could do the customization.
Static-looking URLs: The site should look like a static site. The URLs should be all lowercase and end with .html. It should be possible to modify all the URLs to look as if all pages are static. Apache's mod_rewrite and the URL Rewrite Filter are great tools for making this happen, but it'd be nice if the administration of the application allowed for setting these rules.
It shouldn't look like a CMS: No login links, no registration links, etc.
Ability to easily add dynamic content: It should be easy to add dynamic content - such as RSS Feed headlines to pages.
Menu Customization: In the application, it should be possible to create menus (both main and local navigation) and configure a page to highlight a menu when a particular page is shown.
Versioning of pages: In case someone messes something up, it should be easy enough to revert back to a previous version.
Easy to use: It should be possible to train a marketing person (with little technical knowledge) how to use the system in 10-15 minutes.

For technical companies (such as Virtuas), there are a few additional requirements I'd like to see:

Articles with syntax highlighting: It should be easy to publish articles with code that's colorized. The Java2HtmlPlugin for JSPWiki is a good example of this. I currently don't know of any for XHTML, XML or scripting languages like Ruby or Python.
File Upload: For uploading white papers and other technical publications.

So I've started looking at open source CMS's that fill my requirements. Last weekend, I wrote a solution that fills all of these requirements using SiteMesh, JSPWiki, CSS Framework, Acegi Security and the URL Rewrite Filter. It only took me about 6 hours to complete, but after completing it - I started wondering if I really wanted to start another open source project and maintain it. The answer is no, I don't want to create something new - I want to use something that's already out there. However, since I do have something that satisfies all my requirements, I will use it if I can't tweak an existing OS CMS enough.

Here are a list of CMS's that I'll be looking at in the next week or so. If you're associated with any of these projects, please leave a comment and let me know how many of my requirements you satisfy.

I'm a bit hesitant on Daisy b/c it requires XSLT knowledge for design customization. Magnolia has long URLs and doesn't appear like a static site - and the PHP ones often have .php in their URLs. It should be an interesting investigation to see if these (seemingly) heavyweight solutions can solve a few simple requirements.

[OSCON] Wednesday Morning

Matt Raible — Wed, 3 Aug 2005 11:15:08 -0600

I got a good night's sleep last night so I'd be fresh and ready for the Smackdown today. Matthew Porter, Scott Delap and I visited Jive Software's offices last night and had a great time sipping suds in their beautiful downtown office.

My AppFuse tutorial yesterday was well-received by a packed room of developers. Rather than writing code the whole time and doing a measly 30-slides, I added a bit more meat about Spring, Hibernate and testing. Most of audience was unfamiliar with Spring, so this seemed like the right thing to do. Of course, this led to more talking and less coding, but most of the folks I talked to were nevertheless very happy with the tutorial. If you'd like, you can download my presentation from the event.

Thanks to Rob Harrop and Thomas Risberg for both letting me know about the lack of Spring experience, as well as sitting in on my session. It was pretty cool having these guys in the room, as well as SiteMesh/jMock inventor Joe Walnes. Without these guys, many of the cool features in AppFuse would not be possible.

Now I'm sitting in the beginning Keynote session at OSCON, where they've announced they have a record 2000 attendees this year. In addition, it looks like OSCON is in Portland for the long run - this is the 3rd year it's been in Portland. Rather than moving to a new city like they used to, they've decided to stay b/c conference attendees like it so much.

The "unofficial" tagline of the conference is fun. Open source is fun and exciting - both to develop and use. This is in stark contrast to closed source software that tries to stay stable and boring, with no surprises.

When O'Reilly's CodeZoo launched, it only listed Java open source projects. As of today, they've added python.codezoo.com and ruby.codezoo.com, in addition to java.codezoo.com.

Tim O'Reilly

O'Reilly is not just book publisher or conference producer - but also a company that looks to the future and tries to figure out what's next. To highlight this vision, they've created O'Reilly Radar.

We're currently going through "The Open Source Paradigm Shift". Integration of commodity components has led to a new model where value gets captured. Rather than being at the software level, it's at the services level.

Key Questions for Open Source Advocates

Will "web 2.0" be an open system? What do "open services" look like?
Data as the "Intel Inside" - will we end up needing a Free Data Foundation in 2010?
How does the paradigm shift change our business models and development practices?
Who shoujld we be watching and learning from?

Things on O'Reilly's Radar

Ruby on Rails: new platform and new language. May well be the Perl of Web 2.0.
GreaseMonkey: a Firefox extension that alters websites to fit your view. A website is traditionally closed. GreaseMonkey "opens up" a website and rewrites it for the user.
HousingMaps.com: leveraging Google Maps and existing data from a bunch of different webservices to build a better website.
Ajax: html, javascript and css. The "css" tag on del.icio.us has gone down as the "ajax" tag has gone up.
Findory: Uses the articles you like in blogs and news, and finds similar articles. Similar to Amazon's recommendation system.
Internet Telephony: Asterisk in particular. Skype and Broadvoice. Broadvoice is pushing BYOD (Bring Your Own Device).
HomeBrew: similar to Tivo.
Firefox
Opening up hardware, not just software - i.e. Car PC Hacks, Smart Home Hacks.

For more, go to the Visualizing Technology Trends on Thursday afternoon. For the first time, the computer book market has stabilized. This is a good sign that the computer industry is about to start rebounding. As far as the book market goes, it's market share and growth - Java still leads the pack by a pretty wide margin. A large reason of this is due to Open Source Java. SimplyHired.com spiders 7.9M jobs on 755 different job boards. General books on Linux are up, especially non-RedHat distros. Books on RedHat have decreased significantly.

This is an interesting conference to be at when you're a Java developer. For the most part, everyone seems to be Perl fans, followed by Python, and a few Ruby guys. Most of these developers are very vocal about the fact that they don't like Java. Then again, Java is the leader in many areas - and it's the open source way to hate the guy on top.

Kim Polese, SpikeSource

Building on the Architecture of Participation. A transformation from Do It Yourself (DIY) to Do It Together (DIT). Thanks to the architecture of participation, open source has achieved World Domination - as evident by governments mandating it and IBM pouring billions into it.

The architecture is characterized by:

Commoditization of software
Network-enabled collaboration
Software customizability

In Phase I, we built and we built with. Open source had DIY origins. Now we're in Phase II, where increasingly the action is out in the long tail. Countless new building materials are piling up on the long tail. Now it's possible to build just about anything with anything. IT shops are building a phenomenal set of DIY "packages" that combine components from both ends of the curve.

The two problems with this:

Velocity mismatch: all components are on different release schedules. Linux, Apache, MySQL - all on a different release schedule. In addition, the ones on the other side (Lucene, Struts, Mambo) are on a different cycle.
Dependencies: When one version of one product changes - what happens to all the dependencies?

To solve these problems, companies are developing formalized proceses like review boards, support centers, OSS incubation centers, testing groups and they're certifiying / defining stacks internally. Most of this work is laborius and not related to the core competency of the business.

What's next? Phase III - IT becomes core. They do this by offloading critical but non-strategic work to independent service companies. DIY evolves into DIT with the help of independent service companies. Of course, this is all leading up to the fact that SpikeSource provides these services. It's funny that as soon as Kim said "SpikeSource" - all the presentation screens in the room quit working (not on purpose). A minute later they're back. This goes to show that marketing is not liked by the Open Source Gods.

"Testing is the single biggest refactoring shift in sofware." - J.P. Rangaswami, CIO, DrKW.

We need testing on a massive skale. For this reason, Murugan Pal and Ray Lane started SpikeSource. They saw the next phase is testing open source software so we can scale testing, together. Solve velocity mismatch and dependency problems with rapid per-defect patch management and dynamic stack configuration.

Testing has always been the software's ugly stepchild. We need to scale open source testing the way we scaled open source development. Some perspective: Microsoft has a 1:1 ratio of developers to QA Engineers. There's no Microsoft for open source software, nor should there be. To solve testing on a massive scale, you need participation plus automation. For models of how both scale, think eBay, Google and Amazon. Their best assets were their customers that supplied data that made their services more useful.

Testing is just one service among many. The Linux distros and middleware core building blocks have been there for awhile. Now we have applications and service companies as well. Who benefits when we have abundant integrated, tested, validated automatically patched stacks? IT and ISVs shift high-value development resources to customer-faces - differentiating features and services. In addition, many other groups benefit and higher quality software gets developed.

Testing will do for open source what it did for chip design a generation earlier. Testing is what catapulted the chip industry forward in the 80s. The new testing tools moved VLSI foward. Countless new IC-powered products were made possible and at much faster development speeds. Solving the testing problem can't be done by one company alone. "Come Test with Us..."

After Kim, another speaker (Andrew from OSDL) began his talk. He talked in a monotone and lacked a presentation. The room quickly began to leak people, me being one of them.

[OSCON] Tuesday Afternoon

Matt Raible — Wed, 3 Aug 2005 00:30:10 -0600

Creating Passionate Users
Presented by Kathy Sierra, OSCON 2005

How do you create passionate users? People will do anything and be enthusiastic about it if they're passionate about it. For example, Nikon.com teaches you how to be a better photographer. In their tutorials, they happen to mention that you might need a better camera to take better pictures. If you're going for passion, you have to provide a continuous path for becoming more knowledgeable about a product - and eventually becoming an expert in something. The beauty is the path or thing you provide doesn't even have to be related to the product. It simply has to provide users with enthusiasm about something you provide, which in turn supplies them with the path for that passionate experience.

"Things that look better actually work better." In other words, aesthetics matter.

When writing a book or documentation - you have to make what you're writing about matters to the user. It has to be so important to them that it gives them a queasy feeling. Naturally, the person won't be interested in what you're talking about - you need to be able to communicate the real core of why a person would have an emotional reaction and why it matters to them. "Well, why didn't you say that?" is the reaction you're looking for. If you're can answer that question w/o answering all the questions leading up to id - you're golden. You're supposed to try and scare them to the point that they're never going to have sex again, and then step back one level. What's the compelling meaningful benefit of the product?

You need to seduce your users and keep them interested and passionate by challenging them to learn more. Flow is the feeling when you have no sense of time - and you need to somehow figure out how to get your users into the flow. As long as you believe you're only one compile away from fixing it - you'll spend hours working on something. This is the flow state and comes from the perfect balance between a challenge and the skill+knowledge to solve that challenge.

One of the challenges to creating passionate users is to establish some sort of next level that your users can get to. First of all, you have to figure out what the next level is, followed by what "new powers" and abilities you can give to your users once they get to that level.

Tips for engaging users

When writing, use a conversational tone. The brain has a conversation with text when you read it and you'll have a 40% better retention rate by reading writing with a conversational tone. Also, use pictures whenever possible since they often make things easier to understand. Don't reveal everything - make your users curious.

What do film makers and novelists do? They tell stories.

Where there is community, there is legend, myth, passion and stories. Where there is passion, there are people. How can you propagate the stories and people from the project?

People don't care about you - all they care about is how they feel about themselves after interacting with your product or service.

This was a great session by Kathy and I was very impressed how she presented it. No laptop, just an overhead projector. Lots of group activities and lots of group discussions. I could easily see Kathy and Bert writing a book on Creating Passionate Users.

[OSCON] Monday Afternoon

Matt Raible — Mon, 1 Aug 2005 17:02:31 -0600

Ruby on Rails - Enjoying the ride of programming
Presented by David Heinemeier Hansson, OSCON 2005

About David: started doing Ruby in June 2003. Involuntary programmer of need, served 5 years in PHP. Spent 7 months in a Java shop.

Prerequisites of play: Ruby 1.8.2, dated December 25th. A database, pick one of 6. The RubyGems miner. Some gems called Active and Action.

Directory structure that Rails creates is more for convenience than anything. By picking conventions for you, it makes things easier. It might feel like flexibility is being ripped away from you - but you can change the defaults. However, by following the default settings, things will just work and life will be much easier for you as a developer.

I did a bit of playing on my PowerBook while listening to David's talk. I have Tiger installed, but found that Ruby 1.8.1 was installed on my machine (in /sw/bin/) thanks to Fink. My running "rm -r /sw/bin/ruby" and restarting iTerm, the default changed to /usr/bin/ruby, which is 1.8.2. From there, I downloaded and installed the Rails Installer.

I hate to admit it, but this talk is pretty boring so far. Probably because I've read David's blog for the past 6 months and watched most of the Rails videos. I haven't really learned a whole lot in the first 45 minutes of this talk. To be fair, the content of the talk seems to be properly targeted - there's been a fair amount of questions and everyone seems to be interested. Almost all of the seats are filled in the room; 3-4x as many folks as Dave Thomas's Ruby talk.

One interesting thing I've learned today is many features of Rails (i.e. Webrick) are actually a part of Ruby, not Rails. In addition, Ruby seems to have frequent releases and more features are added to the language each time. I guess that's the advantage of having a language that's not developed by committee.

When creating model objects in Rails, the default is to use a plural form of the object for the database table name. For example, a comment model will map to a comments table. Dave Thomas did mention in this morning's session that Rails isn't smart enough to figure out "sheep" - it gets maps to "sheeps". Apparently, you can easily override this behavior by specifying use_plurals=false somewhere. Another convention built-in to the framework is that the primary key is named "id" and its an auto-incrementing field.

"The database is a data bucket. I don't want any logic in my database, I want it all to be in my data model."

Rails doesn't handle composite primary keys. Rails is mostly designed for green-field development, where you get to control your database and its schema.

There are a number of key properties you can use in your database tables (a.k.a. your model objects) that will automatically get updated if you name them properly. Their names are created_at (datetime), created_on (date), updated_at and updated_on. There are also a number of time-related helpers, i.e. distance_of_time_in_words_to_now(date) » less than a minute ago.

Rails also has the concept of filters, which you can apply to a group of controllers. To use a filter, you define the filter method in controllers/application.rb and then you have to add a before_filter clause in each controller you want it to be applied. While it's cool that Rails has filters, it would be nice if you didn't have to configure the controllers that filters are applied to in the controller. To me, it seems more appropriate to be able to configure the where the filters are applied externally to the controllers. It seems more natural to me that you'd put something like apply_to_controller => { :controller1, :controller2 } in application.rb.

For doing page decoration with Rails (i.e. SiteMesh), you simply create a decorator in views/layouts. If you want a particular decorator to apply to a particular controller, you just name the file the same as the controller's URL. For example, if you have a posts controller (really a PostController.rb file), you'll create a decorator named posts.rhtml to decorate all the HTML rendered from the PostController - regardless of whether you're rendering from a method or from a view template. To have a decorator apply to all controllers, you can simply create a file named view/layouts/application.rhtml. This seems like something that SiteMesh could easily do as well - for example defaulting to /decorators/default.jsp (or something similar).

One thing I like about Rails is it's flash concept and how easy it makes it to display success messages. In my experience with Java web frameworks - many make this more difficult than it should be.

Testing Rails Applications

When running tests, Rails automates the creation of a test database instance that mirrors the schema of your development database. One slick thing in a Rails project's Rakefile is that you can run all the tests that you've touched in the last 10 minutes. I think one of the most unique thing about Rails/Ruby vs. Java is all that almost all of the files (Rakefile, code generation scripts, etc.) are written in Ruby.

Controller tests have a "mini-language" for simulating a browser when testing controllers. For example:

def test_login 
  get :login
  assert_response :success<
  assert_template "login" 
  
  post :login, :password => "secret!"
  assert_response :success
  assert !session[:authorized]
  
  post :login, :password => "secret"
  assert_response :redirect
  assert session[:authorized]
end

In the Controller tests, you can set cookies, parameters and mimic almost everything the browser can do. You can also test that your model objects have been manipulated appropriately. For example:

def test_create_post 
  post :create, :post => { :title => "This is my title", :body => "1" }
  assert_response :redirect
  assert_kind_of Post, Post.find_by_title("This is my title")
  
  post :create, :post => { :title => "", :body => "1" }
  assert_response :success # something was rendered, regardless of error messages
  assert_equal "don't leave me out", assigns(:post).errors.on(:title)
  #or assert_equal 1, assigns(:post).errors.count
end

The find_by_title method is a dynamic finder, where ActiveRecord creates find_by methods for each attribute of the model object. Another cool feature of testing is you can add a line with "breakpoint" in it - and the test will stop executing there - giving you access to all the variables at that point.

Ajax

The main reason for integrating JavaScript into Rails is so developers don't have to write JavaScript. For most developers, writing JavaScript is a pain because of browser incompatibilities and such. Rails ships with 4 JavaScript libraries, including Prototype and Script.aculo.us. It's easy to include the default JavaScript libraries in Rails:

<%= javascript_include_tag :defaults %>

Both the link_to and form_tag methods have a "remote" equivalent (i.e. link_to_remote) that allows you to hook into Ajax, and by defining a :complete callback, you can call fade effects and the like. You can override many of the lifecycle stages of Ajax, but the most common is the :complete callback. In a Controller, it's easy to distinguish Ajax calls from non-Ajax calls using:

if request.xml_http_request?
  # do logic, for example rendering partials
end

Partials seem to be a pretty cool feature in Rails. They're actually just parts of a page that you include in a parent page with render :partial => "viewName". The slick thing about partials is you can actually populate their model and return them in a controller after an Ajax call.

The Ajax demos that David just showed are pretty cool. He was able to easily show how to delete a comment in his "weblog app", as well as add a new comment - w/o refreshing the page. The slick part of the add was he was easily able to add the new comment id to the Ajax response header, and then grab it in a callback and use the id to reference a <div> and use the yellow fade technique to highlight and fade the new comment.

That's the end of Dave's talk, and the first day at OSCON. Thanks to Dave and David for showing me the cool features of Ruby and Rails.

[OSCON] Monday Morning

Matt Raible — Mon, 1 Aug 2005 11:20:05 -0600

Facets of Ruby
Presented by Dave Thomas, OSCON 2005

I'm sitting in Dave Thomas's session on Intro to Ruby at the Oregon Convention Center. It looks like someone finally figured out the main problem with conferences - lack of power outlets. Kudos to O'Reilly - they've put power strips at the base of every table in this room. With the high-speed wireless and unlimited power, this conference is getting off to a great start.

Is programming still fun?

Round about 2000, programming started getting tedious for Dave - after having fun for the past 25 years. When we program, we combine all the problems of the artistic side of the race with all of the problems of the scientific side of the race. The only way to be successful at it is to enjoy doing it.

Is programming productive?

It has to be to enjoy it. The most satisfying thing about programming is watching it run. That's why scripting languages are so great - because you can see it run now. Myth: a good programmer can be a good programmer in any language. Language and tools make a difference - a good programmer knows which language to choose for a particular problem.

This session is not going to be a syntax session. Damn, sounds like I won't really learn how to program Ruby in this session.

Ruby, the Language

Born in Japan in 1994. Father: Invented by Yukihiro Matsumoto (Matz). Mother: Ada, Smalltalk, CLU, Perl, Lisp. Grew very rapidly in 2000, outpaced Python in 2000. Became international star in 2004.

Dave and Andy are language freaks and downloaded Ruby 1.4 shortly after finishing The Pragmatic Programmer. It passed the 5-minute test, the 1/2-hour test and Dave ended up playing with it all morning. The first Pickax book was 500+ pages long, and they wrote it because there wasn't much English-language documentation on Ruby. Ever since Rails, Ruby's adoption has grown exponentially.

Ruby is a multi-paradigm language: procedural, object-oriented, functional, meta-programming. You can write procedural code, but you'll be using OO concepts at the same time. You can do all of these at the same time.

Ruby code example:

# Generate Fibonacci numbers up to a given limit

def fib_up_to(max)
  f1, f2 = 1, 1
  while f1 <= max
    puts f1
    f1, f2 = f2, f1+f2
  end
end

fib_up_to(1000)

Methods start with def and end with end. The parenthesis around the method arguments are optional.

Now Dave is ragging on Java Programmers and how they discount Ruby because of its duck typing. In a Java program, most things are dynamically typed too. This is because most objects are stored in collections and whenever you pull things out of a collection - you have to cast from Object to the real type. The argument is that you don't have to have static typing. Dave hates Generics because he thinks they should've just done automatic casting.

The basic gist of Dave's argument is that we use dynamic typing (using casting) in Java all the time and you don't see Runtime exceptions all of the place. So the biggest proponents of static typing are actually using dynamic typing all of the place.

Back to the code: in Ruby, you don't need parenthesis around conditionals (for instance in the while statement above). The main reason we have parenthesis is because of Fortran. There's no reason for them. You can put parenthesis and semi-colons in your code, but you don't need to. In this code example, the variables are scoped for the duration of the method. puts (pronounced "put s") just prints the value of a variable to the console.

class Song
  def initialize(a_title)
    @title = a_title
  end
  def title
    @title
  end
end

Instance variables in Ruby start with an @ sign. The first time you use them, they spring into existence. If you access an instance variable and it's value hasn't been set - it's value is nil. Using the return keyword at the end of a method is optional - the default is to return the last line of a method. You can change the "title" method to use

attr_reader :title

The attr_reader call is actually a method of Class:class. The attr_reader will dynamically add an accessor (that looks like the title method above). To create a setter, you can use attr_accessor and it'll create both a getter and setter.

Ruby is a single Inheritance language.

class KaraokeSong < Song

  attr_reader :lyric

  def initialize(a_title, a_lyric)
    super(a_title)
    @lyric = a_lyric
  end
end

Unlike Java, the super call can happen in any line, or not at all. To solve the single-inheritance problem, you can use mixins and apply them to any class.

Blocks and iterators are pervasive in Ruby, and look to be very easy to use.

3.times { puts "Ho!" }

hash.each { |key, value|
  puts "#{key} -> #{value}"
end

To do method calls with blocks, you use the yield keyword.

You can use blocks to simplify Resource Management and automatically close resources.

File.open("myfile.dat") do |f|
  name = f.gets
  # whatever
end

When the above code hits end, the file is automatically closed.

Duck Typing

Strongly-typed objects, untyped variables and methods. Types are implicitly determined by the things that an object can do. Duck typing is great for testing, refactoring and maintenance. This is very similar to concepts in Smalltalk. There is a strong commitment to unit testing in Ruby - which makes duck typing even easier to use. Duck typing makes things very easy. For example, you can have a method that takes a file as a parameter - and writes data to it. You can test this method by passing in a String (which also supports << for appending) and verify that your method's logic works.

Duck typing is useful in regular code for reducing coupling and increasing flexibility. Ruby community now differentiates the type (what it can do) of an object and the class (what generated it) of an object.

The Road to Metaprogramming

Metaprogramming is really, really powerful in Ruby. Library writers use it, but most developers don't use it. The ActiveRecord framework is an example of metaprogramming. Rather than being an O/R Mapping tool, it's more of a database table wrapper. The belongs_to, has_one, has_many and other validation_presence_of method calls can be written by you. Allowing you to write DSL (domain-specific languages) that appear to be a part of the Ruby language.

4 steps to metaprogramming:

Classes are open: in Ruby, you can always add methods, attributes, etc. to existing classes. For example, you can add an encrypt() method to the String class. Isn't this dangerous? What if someone changes the logic of + for math expressions. No, because if one of your programmers overrides methods that breaks things - you take them out in the parking lot and beat them with a rubber hose! The language shouldn't prohibit us from doing powerful things.
Definitions are active: code executes during what would normally be compilation. As the class if being defined, the code is being executed. For example, you can check if an environment variable is set - and create methods (i.e. log.debug()) accordingly. This can be great for caching.
All method calls have a receiver: Methods are executed on objects. There's always a current object: self. Methods with no explicit receiver are executed on current object.
Classes are objects too: You can easily add methods to classes.

Many more Ruby features: Reflection and ObjectSpace, Marshalling and Distributed Ruby, Tk, Gtk, Fox, networking, databases, etc. Garbage collection, Threads (like Java green threads), Modules and mixins. ObjectSpace - allows you to reflect on all of the objects that exist at runtime. Marshalling allows you to serialize into binary or text formats. No XML - uses YAML instead. Unlike XML, it's readable and looks more like a properties file. Modules (and their methods) can be easily included into a class simply by using "include ModuleName".

Now Dave is going to write a program to extract book sales ranks from Amazon pages, publish them as an RSS Feed, store them in a database, and access via a web application (using Rails). Since this is likely to involve a lot of live coding, I probably won't blog the code Dave writes.

Web Applications in Ruby can be done with Simple CGI, FastCGI, mod_ruby and frameworks (like Rails). Iowa, CGIKit, Nitro and Ruby on Rails are all web frameworks in Ruby. Iowa is a Ruby implementation of Apple's WebObjects. Dave's quote: "Apple really screwed up with WebObjects, they could've owned the market on web frameworks."

Summary

Use Ruby because it is Lightweight. A Ruby download is under 10 MB. Ruby Gems allows easy package management for downloading libraries and documentation.
Use Ruby is Transparent. It's nice and easy to read - and it only takes a couple of hours to learn its syntax.
Portable - runs on PDAs and Mainframes.
Open Source - MIT/Artistic style license. 1.8.3's regular expression engine is LGPL, 1.9's engine will be BSD-style license.
Easy to Learn - uses the Principle of Least Surprise. Things seem to work as you'd expect. Dave knows people that've downloaded Ruby and put web applications on line in the same morning - w/o any prior knowledge of Ruby.
Fun! It's enjoyable to program in.

Ruby Resources

http://www.ruby-lang.org
http://www.rubygarden.org
http://www.rubyonrails.org
com.lang.ruby - ruby-talk mailing list
http://pragmaticprogrammer.com/titles/ruby (1st edition online for free, covers Ruby 1.6)
Other Books: The Ruby Way, Ruby Developer's Guide, Agile Web Development with Rails (available for the first time at the OSCON bookstore)