Matt RaibleMatt Raible is a Web Architecture Consultant specializing in open source frameworks.

The JHipster Mini-Book The JHipster Mini-Book is a guide to getting started with hip technologies today: AngularJS, Bootstrap, and Spring Boot. All of these frameworks are wrapped up in an easy-to-use project called JHipster.

This book shows you how to build an app with JHipster, and guides you through the plethora of tools, techniques and options you can use. Furthermore, it explains the UI and API building blocks so you understand the underpinnings of your great application.

For book updates, follow @jhipster-book on Twitter.

10+ YEARS


Over 10 years ago, I wrote my first blog post. Since then, I've authored books, had kids, traveled the world, found Trish and blogged about it all.

[OSCON 2008] The State of Lightning Talks

From this session's detail page:

As a repeat of the last 2 years, 15 open source project leaders will be given five minute lightning talk slots to bring the audience up to date on their projects.

This year’s speakers and projects include: Brian Aker: Memcached, Glynn Foster: OpenSolaris, OSI: Danese Cooper, MySQL: Monty Widenius, PostgreSQL: Bruce Momjian, GNOME: Dave Neary, Gentoo: Donnie Berkholz, OpenOffice.org: Louis Suarez-Potts, Jabber: Peter Saint-Andre, Mozdev: Brian King, OpenID: Scott Kveton, Open Scrum: James Dixon, and Cliff Schmidt for the Talking Book Project. A couple more projects may be added later.

Each speaker has exactly 5 minutes, and we use various “fun” tools to make sure they stick to their time. The session usually ends with a “zinger” presentation worth staying for.

Unlike the description above, apparently there's 17 projects instead of 15. This session is 95 minutes long (spans 2 normal sessions). I figure this will be the most challenging event for me to blog, so here we go!

OSI (Open Source Initiative), Danese Cooper
Danese has been on the board of OSI since 2001 and will be on the board for another 3 years. This year, they've been asking themselves "What are the ways to make OSI more effective?" The best answer they've come up with is that they should run OSI more like an open source project (with lots of transparency). Minutes are posted publicly on website and they have a public blog. They're also transferring their archives to a searchable system. They'll be using Trac to do more objective tracking of all the issues that've been brought to their attention.

They're also going to establish (formal) membership. They're going to use Apache as a model and allow Charter, Individual, Organizational and have an Annual Members Meeting. ASF currently has 300 members and these folks are in charge of making decisions for the foundation. There's a good chance you'll recognize all the Charter Members of OSI. The names will be announced in the next couple months (I believe). There will be opportunities to be individual or organizational members.

There's new board members this year with an international focus: Nnenna (Africa), Harshad Gune ...honk... out of time.

Open Scrum, James Dixon
James is the founder and Chief Geek at Pentaho. Pentaho recently took their products open source and they're releasing Open Scrum as an agile methodology for managing open source projects. Open Source is not a methodology, it is a set of principles.

Some agile principles are used in open source, but some are not. For instance, trusting developers and face-to-face contact can be difficult with open source projects. Simplicity sometimes comes out of open source projects, but usually only for projects with a large developer base. There's a 45% good fit between Agile and Open Source.

Open Scrum is based on Scrum, but geared towards open source projects. It's simple and flexible and template driven. It scales from single-geek, part-time, single-projects to massively multi-geek full-on and multi-layered undertakings.

MySQL, Monty Widenius
Monty is the Chief Fonder of MySQL and he's talking about what's new in MySQL. There's been no new releases for users, but internally they've been doing lots of bug fixes in 5.1. He expects to have a reasonable GA of 5.1 in 3-5 months. 5.0 will include online backup and optimizer changes for faster joins and sub queries. 5.1 has an InnoDB Plugin.

MySQL was acquired by Sun for $1 billion and 6 months later, life is good. MySQL Backup was announced to be crippleware by MySQL management, but later changed to be open source.

Switched to use Bazaar and Launchpad to increase community participation. Started public bug hunt for MySQL 5.1. Started using case competition for new features in 5.1. There's also a new MySQL forge. Please help MySQL get better by filing bugs!

Maria engine 1.0 and 1.5; now crashsafe and multi-versioned. Transactional part to follow soon.

OpenID, Scott Kveton
Scott is on the board of the OpenID Foundation and his son is on stage with him (with a Twitter shirt on). In terms of adoption, they have a hockey stick graph with the big upturn starting in February 2007. OpenID is only 3 years old and there's over 20,000 sites that now support it. There's about ~500 million OpenIDs (including every AOL, Yahoo, MySpace and LiveJournal user).

OpenID 2.0 was released as final 12/7/2007. What's next: usability (currently sucks), security (PAPE), integration with other protocols, more than just providing, and possibly OpenID 3.0. One of the biggest problems with usability is people just can't remember URLs, so they're trying to figure out a URL -> e-mail mapping.

OpenID Foundation has added 5 corporate board seats and has 7 community board seats and elections are in August. You can get code at openidenabled.com and you can get involved at openid.net.

Gentoo, Donnie Berkholz
Donnie is one of 7 people that's on the council that tries to make sure Gentoo is going in a good direction. In the last year, they've been struggling with community and releasing. Gentoo is just getting old enough where people are quitting and new developers are coming on board. In 2005, they had a lot of voters for their council. 2006 and 2007 were bad years from a community participation standpoint. This year, they've fixed "poisonous people" and participating is back up.

Rather than voting for single candidates, they use a "rank all the candidates" voting system. Community is very important and they've recently kicked out 3 people that didn't think community was important. They released 2008.0 a few weeks ago. It was supposed to be released in February. When they released, they made the front page of Digg, which made Donnie very happy. Gentoo news announcements were way down in 2007, but now they're back up.

Recommended books for community building: Good to Great, The No Asshole Rule, Getting to Yes.

PostgreSQL, Bruce Momjian
Bruce is a PostgreSQL core developer and works for EnterpriseDB. The future for PostgreSQL is going to be a lot different than the past. PostgreSQL's path to the future isn't a straight line. Rather, it's a meandering path like a road up a mountain with switchbacks. People would like a catapult to put features into to get things in, but it's not that easy. It's difficult to improve software that's been around for 20 years.

Companies have very clearly defined goals that usually revolve around money. With open source, it's a lot different. Goals are usually for a feature set, for adoption of open source or for the challenge and fun of it.

PostgreSQL 8.0 was released in January 2005 and had a lot of enterprise features: Win32, Savepoints, Point-In-Time Recovery, Tablespaces. The latest release (8.3 in February) was much different and much more exotic features. In the future, they're going to expand on running queries on multiple CPUs and maintaining their leadership in the open source database space.

Bazaar, Mark Shuttleworth
Mark's interest in distributed version control predates his interest in Ubuntu. If we can elevate our process of version control, we can elevate the entire open source community. Bazaar is a distributed version control system that's developed in Python. It focuses relentlessly on a couple of key values:

  • Cross-platform: Linux, Windows, Mac OS X
  • Performance: sub-second status on > 10,000 files
  • Being adaptive to the way that people want to work (can use it like Subversion if you want or you can have distributed branches or have a team checkout)
  • Extensibility and embeddable - has a plugin system
  • Python Hackable - people can easily modify

Projects that use Bazaar: GNU, MySQL, Twisted and Ubuntu.

Memcached, Alan "Dormando"
Memcached is a very simple project with a mailing list with very simple people on it that run very simple websites. They haven't had a release in 6 months. Memcached was written by a guy named Fitz that wrote it and left. They've only recently created a developer community that's all over the globe. Alan didn't bring any slides, but he did release 1.2.6 RC1 as he was standing on stage. They also hope to release 1.3.1 this week, but there's not a lot of documentation.

People fork Memcached a LOT, mostly because they need a new backend. They've added a new storage engine that's pluggable to hopefully fix the forking problem. Also, their new binary protocol should make it a lot faster for high traffic web sites.

The Talking Bridge Project, Cliff Schmidt
While the network at OSCON has been slow, the folks that live in rural areas around the world have an incredibly slow means of getting information. Their network is often roads that people travel on to communicate with each other.

They've created a Talking Book device that is an MP3 player that costs $5-10. They're also creating kiosks that people in rural areas can interact with to download new information (i.e. podcasts and other educational content). It also has a microphone and can be found at literacybridge.org. They're creating open source projects to help build the device and content around it. They'd love to have open source developers join their projects and help them make access to information easier for rural societies.

Open Source Lab at Oregon State University, Lance Albertson
Lance is a Kansas native and grew up on a farm. He's a Gentoo developer and joined OSL a year ago. OSL was created in 2003 to utilize "extra" unused bandwidth. They provide managed and co-located hosting for open source software projects. They also provide on-the-job learning opportunities for students in system administration and development.

In addition to hosting, they have a development and outreach side. They have 12 student employees that are given developers/sys admins are given real world practical opportunities. Their work is seen and used by many people and students are given extraordinary amounts of responsibility. They're working on a couple external OSS projects: Oregon Virtual School District and One Laptop Per Child.

Outreach wise, they have an Open Source Education Lab and they've been doing Drupal training for OSU employees. They've also helping organize GOSCON, which is an open source conference for governments.

They currently offer hosting for over 70 OSS projects. Services include, email relays, databases, web hosting, file mirroring and code repositories. Notable clients: ASF, Drupal, kernel.org (master), OpenOffice, Freenode, Gentoo Linux & Debian, phpBB and many others. Datacenter currently has 50 racks; they'll be moving to 75 in the near future.

OpenOffice
They've had 150 million downloads and believe there's around 1 million users. There's 350 committers and most of them work from Sun. Presently, their biggest focus is Desktop. In the future, they'll focus on the Web. The idea is that people will want to use OpenOffice from their mobile devices. ODF is a format that can be expressed by any application. OO 3.0 is about extensions, not bloat. They're adding linkage to a free PIM (Outlook replacement). They're also adding better interoperatbility with other suites and formats (e.g., Microsoft Office 2007). They're adding productivity tools and adding toolkits to make things easier for developers. In the next year, they expect to go way beyond 150 million downloads. They also expect to expand greatly internationally.

Mozdev, Brian King
Brian has been in the Mozilla Community since 1999. Mozdev.org is a community site that provides free hosting for Mozilla applications and extensions since 2000. Established as a non-profit organization in 2004. It has over 250 active projects today and there's over 500 developers discussing issues on the project_owners@mozdev.org list.

Services provided: source code hosting, bug tracking (Bugzilla), communication tools (lists, newsgroups, blogs, forums), file management tools, project tagging, wiki, statistics, permissions system and public planning.

They've experienced some grown pains in the last few years, but they're mostly solved now. They've added developer resources to help develop their site, but they need more. Up until Firefox, they were the #1 place to go for Mozilla projects. They're still working on letting people know that they're around and have projects that are interested. They're looking for new avenues for funding.

Moving forward: they'll be add more items to roadmap, making it easier for less technical folks to host and create extensions, solving the funding issue, and developing deeper relationship with Mozilla (MoFo and MoCo). Quite simply, they want to continue with their goal of supporting developers in order to proliferate Mozilla technologies.

Glen Foster, OpenSolaris
Hey - I had a beer with this guy last night at Kells! Rather than focusing on the older Solaris users, they want to move to focus on getting kids to learn and use it. A couple of months ago, they released 200805 and reduced 5 CDs down to a single live CD. The packaging system is now network based. They changes the default shell to bash and changed to GNU tools as they default as well.

They introduced a new site - opensolaris.com. Since Sun released the source code 3 years ago, they've had a long evolution of getting the source out the door. They were using Teamware and doing much of the development behind the door. Early next month, they're moving to Mercurial and they'll be moving all the source outside the firewall. They'll be using Bugzilla as their bug tracking system.

Similar to Gentoo, they've had quite a few people in their community that don't like change. They're working on fixing this and have had good progress. Several years ago, they created a governance system before they had a community. This was a mistake and their new community-driven governance system should work much better.

GNOME, Dave Neary
GNOME is a desktop environment and set of utilities. It's the default on Ubuntu and Fedora. It's the face of Linux on the Desktop. It's also a platform for development. They have a vibrant ISV community that includes Adobe, VMWare and IBM. In addition to desktop, GNOME is also a mobile platform. They also act as a big tent that provides an infrastructure for projects that are part of the GNOME ecosystem.

The most important thing to Dave is the shared vision of universal access to anyone and everyone. Very important things to the GNOME project: usability, internationalization and localization, accessibility (Sun and IBM have done a lot of work). Accessibility is something they've really focused on in the last year. They've created an Outreach Project for Accessibility and have gotten funding from Mozilla, Canonical and Google. Mozilla has done a lot to fund them and help provide accessibility for Rich Internet Applications. To see if a website is accessible, try it in lynx.

Subversion, John Mark Walker
John is the Community Manager for openCollabNet and is not a Subversion developer. Subversion 1.4 was released in September 2006. There was much anticipating for 1.5 and was released last month. The people rejoiced.

What's new in 1.5? Merge tracking: easier to merge changes from trunk to branch and vise versa. You can also do sparse checkouts (replaces -N). Interactive conflict resolution so you can do it from the command line client. It also add changelist support. svn:externals is no longer required to be absolute (sweet!) and you can add peg revisions in your URLs. FSFS repos never change a rev after written to disk. Clients many now perform chained copy/move operations locally. It also supports moving multiple sources for copy and move. Client operations are now significantly more responsive to canceling operations. "resolve" subcommand replaces "resolved" (deprecated). Delete (remove) now takes a --keep-local option.

Now it's time for the fun stuff...

Ken Drachnik had to leave for an emergency and is unable to give his talk on GlassFish. An audience member (Carl Fogel) volunteers to do his presentation. It's very funny, especially since he's not a Java Developer and has never heard of GlassFish. The only thing I got out of this presentation (besides lots of laughs) is that v3 will be released in June 2009.

Drizzle, Brian Aker
Stored procedures, triggers, prepared statements and many other things have been discarded. "One CPU to rule them, and in the query cache, bind them." If you have a query cache turned on in our database, turn it off or delete it.

Drizzle is currently ~420K. The Master Plan is to rethink everything and not assume everything was bad. Also, they can reuse many other libraries w/o writing them from scratch. They don't have to ship every library because many are present on operating systems. In with the new: C99, Posix, package-lib. They're moving to a MicroKernal design and moving code to the edge. No new features will be in core. To add interfaces you have to remove code so there's an equal amount of code.

Multi-Core support: no new locks, remove old locks. Think today (burn memory, messages scales). They can leverage Sun because they can build machines with more cores than anyone else.

Field Types: Less primitives - now there's just one "blob". Removed the 3 byte int and add UUID/IPV4(6). They dumped ACL because we authenticate in clouds. KISS.

You can get involved with Drizzle at http://launchpad.net/drizzle (bzr branch lp:drizzle) and http://launchpad.net/drizzle-discuss (mailing list).

Posted in Open Source at Jul 24 2008, 01:34:08 PM MDT 1 Comment

[OSCON 2008] Web Frameworks of the Future: Flex, GWT, Grails and Rails

Below is the presentation I'm delivering at OSCON today. Unfortunately, I had to remove slides on GWT and Flex to fit w/in the 45 minute time limit. I hope to expand this presentation in the future, as well as continue to develop the side project I'm working on using these technologies.

Posted in Open Source at Jul 23 2008, 04:25:23 PM MDT 19 Comments

[OSCON 2008] Caching and Performance: Lessons from Facebook by Lucas Nealan

Facebook is not just a Social Networking Site, they're a "Social Utility". They have the 4th most trafficked site in the world and over 90 million active users. Of those users, the average usage is 50 pages per day. There's currently over 24,000 platform applications. There's thousands of Apache web servers and hundreds of MySQL and Memcached servers.

The biggest problem with scaling for Facebook is the complexity. Connecting to all the databases is impossible. They have a very large codebase - their homepage has 500 library files and 10,000 functions. Scaling affects resources, particularly with regards to memory consumption and socket connection limits. Cache retrieval is ~10% cpu-user of most pages.

Caching Layers: $GLOBALS, APC, Memcached, Database, Browser Cache, Third Party CDN.

The Globals Cache is a PHP function called "cache_get". The Globals Cache works nicely in that it avoids calling APC and Memcached, but it still requires the overhead of a function call. APC (Alternative PHP Cache) is used for opcode caching (hundreds of included libraries, thousands of functions) and variable caching (hundreds of MB's of data). They use the APC for non-user specific data: network/school information, database information, useragent strings, hot application data, site variables and language strings (now the largest consumer of memory). They don't use it for User data because they don't send users back to the same server each time.

Friends page with a normal run takes 4050ms, with APC enabled it takes 135ms. You can also set apc.stat=0 to gain even more speed (128ms). To bust client-caching, they use APC+SVN with the SVN tag on the file and get the latest version from SVN and store it. Of course, this is a "prime the pump" thing that doesn't happen in production at runtime.

The next layer of caching is Memcached. Facebook currently utilizes over 400 instances of Memcached and has made contributions back to the project. They use Memcached for user-specific data: long profile, short profile, friends and applications. They don't use the timeout feature, but rather use cache invalidation on SQL insert and update. It's harder to do when writing your application, but it's easier to maintain in the long run. To make Memcached faster, they created a PHP extension that reduced PHP function calling overhead and allowed UDP support. The Memcached extension runs ~10% faster realtime than in PHP space.

Facebook likes for each page to render in under 250ms on the backend. To see how long a page took to load, you can mouseover the copyright at the bottom of the page, and a tooltip will show you the elapsed time.

This presentation is available online at http://sizzo.org/talks.

Posted in Open Source at Jul 23 2008, 04:18:15 PM MDT 1 Comment

[OSCON 2008] Google XML Pages (GXP) by Harry Heymann and Laurence Gonsalves

GXP is a templating system that Harry and Laurence developed at Google. It was original created by Laurence in late 2001 (Java run-time, compiler written in Python) as part of the AdWords rewrite. I'm attending this session because I heard from a Google employee that they were using WebWork + their proprietary templating framework for the view. My suspicion is that GXP is that framework. The presentation I'm listening to is available at the following URL:

http://docs.google.com/Present?docid=dcbpz3ck_8gphq8bdt

Google XML Pages has the following features:

  • static type checking
  • convenient parameter passing/modularization system
  • partial markup validation
  • automatic escaping of untrusted content
  • encourages functional style, discourages side-effects
  • internationalization support
  • lightweight runtime

GXP is an open source project as of today and is available at http://gxp.googlecode.com. It's used by AdWords, AdSense, Checkout, Blogger, Analytics, Reader and many more.

HelloWorld.gxp:

<gxp:template name='com.google.sample.HelloWorld'
              xmlns:gxp='http://google.com/2001/gxp'>
Hello, World!
</gxp:template>

Output:

Hello, World!

GXP has compile-time markup validation as well as static-type checking. GXP has native data types: for text/html, text/plain, text/css and text/javascript. It supports loops, conditionals, abbreviations, internationalization (i.e. <gxp:msg>Hello, World</gxp:msg>) with placeholders. You can call GXP in the same package using <call:GXPName>. To call something outside a package, you can use <gxp:import> to import packages or classes. You can also use a qualified XML namespace to access another package.

Posted in Open Source at Jul 23 2008, 03:23:44 PM MDT 3 Comments

[OSCON 2008] An Introduction to Ruby Web Frameworks by Ryan Carmelo Briones

Ryan is a Server Monkey / Code Sumari for Edgecase, LLC in Columbus, Ohio. A framework allows you to create re-usable code. Frameworks allow you to use encapsulation. Frameworks tend to be domain specific. For example, Rails works really for CRUD application, but not for others (i.e. Twitter).

Why Ruby?
Ruby has been Object Oriented since day 1. Ruby promotes Beautiful Code that's easy to read and maintain. Yes, MRI has performance issues. Matz has said "I'm a language designer" and has turned over the VM to others for Ruby 1.9. Another thing that might keep folks from using Ruby or its web frameworks is the libraries available. This is understandable, but it's being solved by alternative implementations. This includes YARV (the official 1.9 implementation), JRuby, IronRuby (not ready for production) and MagLev.

Rack
A framework that provides a minimal API for connecting web services and web frameworks. As a web application developer, this framework allows us to know about web services, but not worry about the details of talking to it. Below is a very simple Rack application.

class HelloWorld
  def call(env)
    [200, {"Content-Type" => "text/plain"}, ["Hello World!"]]
  end
end

Rack allows the handlers do the work and not worry about the web server abstraction. Handlers exist for WEBrick, LightSpeed, Mongrel, Fast-CGI and many others. As an application developer, it allows you to choose different architectures (threaded, evented, etc.). Ryan is talking about Rack first because it's used in all the other Ruby web frameworks.

Ruby on Rails
Rails is 4 years old now and was written by DHH when he was a contractor at 37Signals. Rails doesn't define and grand new ideas, everything has been done before (MVC, code generation, etc.). What Rails did is package everything in a unique way that makes it very easy to use. Rails has influenced a lot of what has come from web frameworks in the last few years. One of Rails' nicest feature is code generation. Ryan showed part of DHH's Create a weblog in 15 minutes video to demonstrate code generation. He noted that the minute he showed was picked because David said "Whoops!" and "Look at all the things I'm not doing". Rails popularized Convention over Configuration using naming conventions and load paths. While this is definitely a cool feature, I think most web frameworks have adopted CoC by now. Maybe not JSF, but who wants to use JSF w/o a framework on top of it anyway?

One warning about Rails: "The Golden Path" can get in your way. Rails is very Opinionated Software and that's how Rails works. As long as you follow that, you should be very productivity. If you decide to go off the Rails (i.e. namespaces), it can be difficult.

Rails uses a DSL in its models (i.e. has_many, has_one for relationships) and in the Rails router. It allows you to very simply map a URL to a controller/method. In addition to DSLs, Rails has first-class testing and its generators create stub tests for you.

Bad things about Rails: too much magic, moves to fast (too many releases).

Merb
Merb was originally developed by Ezra Zygmuntowicz to run alongside a Rails app to handle file uploads. It grew from there and became it's own beast. Merb is very much about using only what you need. It has "package repos" that allow you to add additional features. For example, merb-core doesn't contain an ORM framework, just a web framework. Merb also allows you to choose your ORM. It's standardized on Rack, so it can run on just about any web server. It also included "deferred actions" that allow you to send some URLs to evented web servers and others to threaded web servers. Merb eschews the "magic" that Rails has. It tries to stay away from making it's code a "monument to personal cleverness". Simple code scales better and runs faster.

One of the downers to Merb is that it's flexibility allows you to get down to the nitty gritty. However, it can be less productive than Rails because of its flexibility. Another downside is its documentation and examples are sparse. Merb is not recommended if you're just getting into Ruby.

Camping
Camping is a micro framework (< 4K) developed by why the lucky stiff. It's designed to develop small applications. You can do everything in one file and create prototypes very quickly. It uses Markaby to write HTML code in a builder-style fashion.

Since a Camping application is all in one file, it can be difficult to develop large applications. The solution is to write small apps and mount them in the same URL space. The only issue with small apps sharing the same space is they have to use the same database. One downside to Camping is there is no standard test framework. Mosquito was developed as a solution, but doesn't seem to be maintained.

Sinatra
Simple. Fast. Effective. It's designed to allow creating REST applications with minimal dependencies. Similar to Camping, it has one file for the entire application. Unlike Camping, Sinatra doesn't follow MVC conventions, so it may be difficult to port a Sinatra application to another framework.

Posted in Open Source at Jul 23 2008, 12:33:08 PM MDT 1 Comment

[OSCON 2008] The Keynote

This morning, I woke up awful early to polish my presentation, walked to the train station and rode Amtrak from Salem to Portland. The commute was great: there's nothing better than traveling with power and an EVDO card + the option to get a cup of coffee. After getting off the train, riding The Max and walking to the Oregon Convention Center, I'm now sitting in the Keynote at OSCON. Here's my notes from this session.

10 years ago, leaders of the free software movement got together and tried to figure out a way to help people understand how to get access to software freedom. As they talked, there was a gradual meeting of minds. Finally, one person suggested "Open Source". A few weeks later, there was a larger meeting of people and they heard about this term. It was an idea that changed the idea of software freedom and what free software was. We've come along way since then. Last year, we heard about open source and and it trying to find identity in corporations. This year, we're hearing about corporations trying to find their identity in open source.

The official tag for this conference is: oscon08.

Tim O'Reilly
While this is the 10th Anniversary of OSCON, it's also the 12th Anniversary of the O'Reilly Perl Conference (where it all started). Tim began his activism with Perl when it got on the web. He was thinking about the internet and the online world, from the beginning (when many others were coming from Linux). Open Source was almost named "SourceWare". Tim believes his biggest contribution is bringing Open Source and the Internet together.

"Keep your History" - make the things you put online accessible for years to come.

When OSCON first started, it was all about the OS Wars. Tim is showing a shirt with the famous Ghandi quote on it about "First they laugh at you..." and it has a Linux logo on the bottom. It's seems ironic that Microsoft is now one of the major sponsors of this conference (my thoughts, not Tim's).

Open Source Technology in the Enterprise. IT jobs are 2.3% of all jobs posted, according to the Bureau of Labor Statistics. Technology oriented companies (e.g., Google, Yahoo, Sun) make heavy use of Open Source (40% of all jobs posted by Y!). Open source is growing faster in non-tech companies. Of the open source technologies in the enterprise, the highest share of jobs is Linux (19%), followed by Perl, JavaScript and PHP. As far as the fastest growing, Django and Alfresco are at the top.

Three Big Challenges and Opportunities:

  1. Cloud Computing
  2. The (Open) Programmable Web
  3. Open Mobile

Cloud Computing: Amazon Web Services, Google App Engine, The Engine Yard, etc.

Jesse Vincent: "Web 2.0 is Sharecropping"

Danny O'Brien: "If we want people to have the same degree of user autonomy as we’ve come to expect from the world, we may have to sit down and code alternatives to Google Docs, Twitter, and EC3 that can live with us oon the edge, not be run by third parties."

Basically, Tim is saying the that cloud computing is great, but it doesn't fit well with open source. This is primarily because if you build on a cloud, you have to be careful not to get locked into that platform.

Data is the "Intel Inside".

The Web is the Internet Operating System - the subsystems will be data subsystems.

Locking in data: iTunes and Amazon's Kindle. On the other hand you have Yahoo's BOSS, which is doing the opposite.

We Need the Open Web Platform! Tomorrow's Keynote, "Supporting the Open Web" will talk much more about this.

The Mobile Web has caused the "browser wars" to resurface. However, big companies like Google are putting a stake in the ground and saying "We believe in open". Net Neutrality and The Open Handset Alliance are two of Google's smartest strategic decisions. They understand how much they depend on the open ecosystem.

When we look at our success in the last 10 years, we can be really excited. But what's really impressive is how much we (as an open source community) is how we've risen to new challenges and challenged the openness of new platforms and industries.

Christine Peterson
Christine is the President of the the Foresight Institute. Christine was the person who suggested the term "open source" at the meeting referenced above. Unfortunately, my first battery died as Christine was coming on stage, so I missed writing down the first 10 minutes of her 15 minute talk. She's talking about the openness vs. privacy of keeping US citizens safe. She started her talk apologizing for the ethnocentricity of her talk and moved to quickly note that the e-voting controversy wouldn't have happened if open source software was used.

"Who would have guessed that the folks with the pocket protectors would turn out to be the ones with the right stuff?" -- LA Times

Founding Geeks: Thomas Jefferson (mechanical geek) and Thomas Edison (electricity geek).

You can't just complain about things. The fear is real. We can't just complain about how DC is solving problems, we have to step up and solve them ourselves.

"No Secret Software for Public Sensing Data."

Dirk Hohndel
Dirk is the Chief Linux and Open Source Technologist at Intel. He's talking about Moblin: Linux for Next Generation Mobile Internet. This sounds like something that has been talked about a million times before. Why is it interesting today? Because we're at an open source conference and open source is what makes it interesting.

When people look at Intel, they don't think of open source. However, Intel is very involved in open source and uses an open source methodology internally for their development process. They also have one of the largest grids powered by open source (~100K Linux servers).

Moblin is about the internet, about mobility, about flexibility and extensibility. What's happening today is the ideas of 10 years ago have become affordable to produce (for manufacturers) and purchase (for consumers). There's lots of proprietary ways to develop the mobile web, but it needs to be open in order to prevent lock-in (to a platform) and encourage innovation.

A year ago, Intel started Moblin. Initially, there wasn't a lot of interest from open source developers. The majority of interest came from companies, particularly hardware vendors. To Dirk, this was disappointing as he really wanted a community to guide the project and make choices about the platform. There's lots of Open Mobile/Linux efforts out there, but there aren't any that are truly open - with access to the source code and everything else you'd expect from an open source project. Intel was hoping to announce a cutting-edge infrastructure for Moblin here at OSCON, but they're a few weeks behind. They hope to be ready for soon.

"The hope that I have is the community takes this from us. Show us where to go. Show us where not to go. Help us get this right."

Tim O'Reilly interviews Monty Widenius and Brian Aker
Tim asks how it's going 6 months in. Monty responds that he's very happy they didn't have to go public and that Sun is still trying to figure out what they bought. One of the things difficulties they've seen about encouraging Sun's engineers to be involved in open source is some are hesitant about open sourcing their code. The biggest problem is engineers are afraid of the feedback/scrutiny that their code will receive.

MySQL was very unique as a company in that it was a virtual company, with most engineers working out of their homes. MySQL has become an enabling force for moving Sun to a similar model.

Monty is working on Maria (new storage engine) and Brian is working on Drizzle (a slimmer version of MySQL). Drizzle was inspired by a conversation when Brian was talking to Rackspace's CTO.

"Do less and then create extensibility mechanisms." -- Tim O'Reilly

Posted in Open Source at Jul 23 2008, 11:29:21 AM MDT Add a Comment

OSCON: Where are the good parties at?

Portland, Oregon It seemed to work pretty well for JavaOne, so let's try it for OSCON...

This week, the best Open Source Conference will take place in the beautiful city of Portland, Oregon. I went to high school just south of Portland (Salem) for a couple years and my parents still live there. Portland is one of my favorite cities, my parents are two of my favorite people and OSCON is one of my favorite conferences. Seems like an excellent combination for a really fun week.

I'll be flying to Portland with Jack and Abbie on Tuesday. Wednesday I'll be at the show speaking; Thursday I'll be there listening and likely blogging. I hope to catch a happy hour on at least one of those nights. Friday, my family and I will be heading to the Oregon Brew Fest followed by a night at The Kennedy School.

So when and where are the good parties at during OSCON 2008? Here's what I know about so far - I'll add to this list as comments start flowing in:

As far as finding me at the show, it's probably easiest to follow me on Twitter.

Update: A reader e-mailed me about http://upcoming.yahoo.com/tag/oscon. This seems to be an excellent source of OSCON party information. I've updated the above list with the ones that look good on Wednesday and Thursday.

Posted in Open Source at Jul 20 2008, 12:28:39 PM MDT 2 Comments

Apache 2 on OS X: Configuring mod_proxy and SSL

I recently had to setup Apache as a front-end web server for multiple backend servlet containers. The backend containers serve up different web applications, and the Apache front-end unites them from a hostname and port standpoint. The following instructions describe how to configure Apache 2 on Mac OS X to proxy requests to Tomcat or Jetty running on localhost:8080. It also shows how to enable SSL on Apache and force it for certain URLs in your Java web application.

Apache comes pre-installed on OS X, so you should be able to start it by enabling "Web Sharing" in System Preferences > Sharing.

$APACHE_HOME on Leopard is /etc/apache2. On Tiger, it's /etc/httpd. If you've upgraded Tiger to Leopard, it's likely you'll have both directories so make sure you're modifying the right one. I lost a few hours figuring this out, so hopefully this knowledge will appease some googler in the future.

Configuring mod_proxy

  1. Open $APACHE_HOME/httpd.conf and add the following on line 480 - at the very bottom, just before "Include /private/etc/apache2/other/*.conf".
    #
    # Proxy Server directives. 
    #
    <IfModule mod_proxy.c>
        ProxyRequests On
        ProxyPreserveHost On
    
        ProxyStatus On
        <Location /status>
            SetHandler server-status
    
            Order Deny,Allow
            Deny from all
            Allow from 127.0.0.1
        </Location>
    
        ProxyPass    /myapp    http://localhost:8080/myapp
    </IfModule>

    ProxyPreserveHost allows request.getServerName() and request.getServerPort() to work as if there is no proxy server in place. In other words, even though Tomcat is running on 8080, request.getServerPort() will return 80.

  2. The most important line is the last one as this is the dictates the location of your applications. Add more lines as you need to add more applications.
  3. If everything is configured correctly, you should be able to run sudo apachectl restart and navigate to http://localhost/status. If you receive a "forbidden" error, make sure your /etc/hosts has an entry mapping 127.0.0.1 to localhost (as one of the last entries), or change "Allow from 127.0.0.1" to "Allow from localhost". If you get a "Server not found" error, you can tail the error log at "/var/log/apache2/error_log".

One issue I've seen with mod_proxy is when a request comes in and the backend server is down. When this happens, Apache returns a 503 Service Temporarily Unavailable and it doesn't seem to go away after the backend server is restarted. It does resume proxying after a while, but I haven't determined what causes the proxy to come back to life. If you know a setting that forces mod_proxy to check for the backend server on every request, please let me know.

Configuring SSL

  1. Open $APACHE_HOME/httpd.conf and uncomment the following on line 470:
    Include /private/etc/apache2/extra/httpd-ssl.conf
  2. Open $APACHE_HOME/extra/httpd-ssl.conf and change line 78 to:
    ServerName localhost:443
  3. In httpd-ssl.conf, change line 99 to:
    SSLCertificateFile "/private/etc/apache2/ssl.key/server.crt"
  4. In httpd-ssl.conf, change line 107 to:
    SSLCertificateKeyFile "/private/etc/apache2/ssl.key/server.key"
  5. In httpd-ssl.conf, add the following after SSLEngine on to allow proxying via HTTPS:
    SSLProxyEngine on
  6. Follow the Using mod_ssl on Mac OS X tutorial. For "Common Name/Server Name", use "localhost". You can download the source for mod_ssl (which you need at one point during the tutorial) at http://www.modssl.org/source/.
  7. Run sudo apachectl restart and go to https://localhost. If you get a "Server not found" error, run sudo apachectl -t to verify the syntax of your config files or tail -f /var/log/apache2/error_log to verify there are no errors in the log files.

Forcing HTTPS for certain URLs
If you proxy requests from /myapp -> http://localhost:8080/myapp, request.isSecure() will return false. If you change it to /myapp -> https://localhost:8443/myapp, request.isSecure() will return true. I needed to figure out a way to have http://localhost/myapp go to http://localhost:8080/myapp and https://localhost/myapp to go http://localhost:8443/myapp. Even better, I wanted to configure things in a way so request.isSecure() returned the value based on the originally requested URL, not on the proxied URL. Configuration like the following would be ideal:

ProxyPass    http://*/myapp    http://*:8080/myapp
ProxyPass    https://*/myapp   https://*:8443/myapp

The solution I came up with is to standardize on secure URLs in my application. That is, use /secure/* as a prefix for all URLs that should be accessed via SSL. To follow this convention and force it, I added the following in my application's web.xml file:

<security-constraint>
  <web-resource-collection>
    <web-resource-name>Secure Area</web-resource-name>
    <url-pattern>/secure/*</url-pattern>
  </web-resource-collection>
  <user-data-constraint>
    <transport-guarantee>CONFIDENTIAL</transport-guarantee>
  </user-data-constraint>
</security-constraint>

Once this is in place, accessing http://localhost/myapp/secure/index.html will result in an error. Accessing it using https will succeed. Following this, you can change your ProxyPass rules to the following and all requests to /secure/* will be https; other requests will be sent to http. The order of the rules below is important.

ProxyPass    /myapp/secure   https://localhost:8443/myapp/secure
ProxyPass    /myapp          http://localhost:8080/myapp

If this isn't a good strategy for you, Tomcat has the ability to use a redirectPort (in server.xml) that auto-redirects from http to https when CONFIDENTIAL is used in web.xml. I'm not sure if this redirect will carry through values from a form post.

Posted in Open Source at Apr 24 2008, 10:58:03 AM MDT 8 Comments

History Meme

From my 17" MacBook Pro that sits at home and I don't use much on a day-to-day basis:

powers:~ mraible$ history | awk '{a[$2]++}END{for(i in a){print a[i] " " i}}' | sort -rn | head
148 mvn
71 cd
46 sudo
40 ls
27 java
23 ll
17 svn
16 echo
13 vi
10 mate

From my 15" MacBook Pro that I use on a daily basis:

mraible-mn:~ mraible$ history | awk '{a[$2]++}END{for(i in a){print a[i] " " i}}' | sort -rn | head
83 cd
81 ls
79 sudo
65 mvn
39 mate
28 svn
19 vi
15 rm
12 open
7 tail

Nothing too exciting.

Posted in Open Source at Apr 16 2008, 10:58:45 AM MDT 2 Comments

Proposal accepted for OSCON 2008

OSCON 2008 From an e-mail I received earlier this afternoon:

We are pleased to accept the following proposal for OSCON 2008.

* Web Frameworks of the Future: Flex, GWT, Grails and Rails

It has been scheduled for 16:30 on 23 Jul 2008.

My Abstract:

What if the choices in web framework was reduced to 4? If RIA are the way of the future, it's possible that these 4 frameworks are the best choices for this development paradigm. This session will explore these frameworks, as well as entertain many other's opinions on the future of web development.

RESTful backends are easy to create with both Rails and Grails. Ajax frontends are simple to create and maintain with GWT. Flex gives you flash and a pretty UI. If you're an HTML developer, Rails allows you to quickly develop MVC applications. If you're a Java Developer, GWT + Grails might be a match made in heaven. This session is designed to help you learn more about each framework and decide which combination is best for your project.

I'm really looking forward to learning about GWT and Flex in the coming months. If you have any experience (or opinions) about the abstract above, I'd love to hear it. The louder the better.

For those who haven't been, OSCON is one of those truly special conferences. Possible reasons:

I'm going for all 4 reasons and even made a reservation to stay at The Kennedy School. Should be a fun show.

Posted in Open Source at Mar 17 2008, 07:21:10 PM MDT 9 Comments