Matt RaibleMatt Raible is a Web Architecture Consultant specializing in open source frameworks.

The JHipster Mini-Book The JHipster Mini-Book is a guide to getting started with hip technologies today: AngularJS, Bootstrap, and Spring Boot. All of these frameworks are wrapped up in an easy-to-use project called JHipster.

This book shows you how to build an app with JHipster, and guides you through the plethora of tools, techniques and options you can use. Furthermore, it explains the UI and API building blocks so you understand the underpinnings of your great application.

For book updates, follow @jhipster-book on Twitter.

10+ YEARS


Over 10 years ago, I wrote my first blog post. Since then, I've authored books, had kids, traveled the world, found Trish and blogged about it all.

[OSCON 2008] Web Frameworks of the Future: Flex, GWT, Grails and Rails

Below is the presentation I'm delivering at OSCON today. Unfortunately, I had to remove slides on GWT and Flex to fit w/in the 45 minute time limit. I hope to expand this presentation in the future, as well as continue to develop the side project I'm working on using these technologies.

Posted in Open Source at Jul 23 2008, 04:25:23 PM MDT 19 Comments

[OSCON 2008] Caching and Performance: Lessons from Facebook by Lucas Nealan

Facebook is not just a Social Networking Site, they're a "Social Utility". They have the 4th most trafficked site in the world and over 90 million active users. Of those users, the average usage is 50 pages per day. There's currently over 24,000 platform applications. There's thousands of Apache web servers and hundreds of MySQL and Memcached servers.

The biggest problem with scaling for Facebook is the complexity. Connecting to all the databases is impossible. They have a very large codebase - their homepage has 500 library files and 10,000 functions. Scaling affects resources, particularly with regards to memory consumption and socket connection limits. Cache retrieval is ~10% cpu-user of most pages.

Caching Layers: $GLOBALS, APC, Memcached, Database, Browser Cache, Third Party CDN.

The Globals Cache is a PHP function called "cache_get". The Globals Cache works nicely in that it avoids calling APC and Memcached, but it still requires the overhead of a function call. APC (Alternative PHP Cache) is used for opcode caching (hundreds of included libraries, thousands of functions) and variable caching (hundreds of MB's of data). They use the APC for non-user specific data: network/school information, database information, useragent strings, hot application data, site variables and language strings (now the largest consumer of memory). They don't use it for User data because they don't send users back to the same server each time.

Friends page with a normal run takes 4050ms, with APC enabled it takes 135ms. You can also set apc.stat=0 to gain even more speed (128ms). To bust client-caching, they use APC+SVN with the SVN tag on the file and get the latest version from SVN and store it. Of course, this is a "prime the pump" thing that doesn't happen in production at runtime.

The next layer of caching is Memcached. Facebook currently utilizes over 400 instances of Memcached and has made contributions back to the project. They use Memcached for user-specific data: long profile, short profile, friends and applications. They don't use the timeout feature, but rather use cache invalidation on SQL insert and update. It's harder to do when writing your application, but it's easier to maintain in the long run. To make Memcached faster, they created a PHP extension that reduced PHP function calling overhead and allowed UDP support. The Memcached extension runs ~10% faster realtime than in PHP space.

Facebook likes for each page to render in under 250ms on the backend. To see how long a page took to load, you can mouseover the copyright at the bottom of the page, and a tooltip will show you the elapsed time.

This presentation is available online at http://sizzo.org/talks.

Posted in Open Source at Jul 23 2008, 04:18:15 PM MDT 1 Comment

[OSCON 2008] Google XML Pages (GXP) by Harry Heymann and Laurence Gonsalves

GXP is a templating system that Harry and Laurence developed at Google. It was original created by Laurence in late 2001 (Java run-time, compiler written in Python) as part of the AdWords rewrite. I'm attending this session because I heard from a Google employee that they were using WebWork + their proprietary templating framework for the view. My suspicion is that GXP is that framework. The presentation I'm listening to is available at the following URL:

http://docs.google.com/Present?docid=dcbpz3ck_8gphq8bdt

Google XML Pages has the following features:

  • static type checking
  • convenient parameter passing/modularization system
  • partial markup validation
  • automatic escaping of untrusted content
  • encourages functional style, discourages side-effects
  • internationalization support
  • lightweight runtime

GXP is an open source project as of today and is available at http://gxp.googlecode.com. It's used by AdWords, AdSense, Checkout, Blogger, Analytics, Reader and many more.

HelloWorld.gxp:

<gxp:template name='com.google.sample.HelloWorld'
              xmlns:gxp='http://google.com/2001/gxp'>
Hello, World!
</gxp:template>

Output:

Hello, World!

GXP has compile-time markup validation as well as static-type checking. GXP has native data types: for text/html, text/plain, text/css and text/javascript. It supports loops, conditionals, abbreviations, internationalization (i.e. <gxp:msg>Hello, World</gxp:msg>) with placeholders. You can call GXP in the same package using <call:GXPName>. To call something outside a package, you can use <gxp:import> to import packages or classes. You can also use a qualified XML namespace to access another package.

Posted in Open Source at Jul 23 2008, 03:23:44 PM MDT 3 Comments

[OSCON 2008] An Introduction to Ruby Web Frameworks by Ryan Carmelo Briones

Ryan is a Server Monkey / Code Sumari for Edgecase, LLC in Columbus, Ohio. A framework allows you to create re-usable code. Frameworks allow you to use encapsulation. Frameworks tend to be domain specific. For example, Rails works really for CRUD application, but not for others (i.e. Twitter).

Why Ruby?
Ruby has been Object Oriented since day 1. Ruby promotes Beautiful Code that's easy to read and maintain. Yes, MRI has performance issues. Matz has said "I'm a language designer" and has turned over the VM to others for Ruby 1.9. Another thing that might keep folks from using Ruby or its web frameworks is the libraries available. This is understandable, but it's being solved by alternative implementations. This includes YARV (the official 1.9 implementation), JRuby, IronRuby (not ready for production) and MagLev.

Rack
A framework that provides a minimal API for connecting web services and web frameworks. As a web application developer, this framework allows us to know about web services, but not worry about the details of talking to it. Below is a very simple Rack application.

class HelloWorld
  def call(env)
    [200, {"Content-Type" => "text/plain"}, ["Hello World!"]]
  end
end

Rack allows the handlers do the work and not worry about the web server abstraction. Handlers exist for WEBrick, LightSpeed, Mongrel, Fast-CGI and many others. As an application developer, it allows you to choose different architectures (threaded, evented, etc.). Ryan is talking about Rack first because it's used in all the other Ruby web frameworks.

Ruby on Rails
Rails is 4 years old now and was written by DHH when he was a contractor at 37Signals. Rails doesn't define and grand new ideas, everything has been done before (MVC, code generation, etc.). What Rails did is package everything in a unique way that makes it very easy to use. Rails has influenced a lot of what has come from web frameworks in the last few years. One of Rails' nicest feature is code generation. Ryan showed part of DHH's Create a weblog in 15 minutes video to demonstrate code generation. He noted that the minute he showed was picked because David said "Whoops!" and "Look at all the things I'm not doing". Rails popularized Convention over Configuration using naming conventions and load paths. While this is definitely a cool feature, I think most web frameworks have adopted CoC by now. Maybe not JSF, but who wants to use JSF w/o a framework on top of it anyway?

One warning about Rails: "The Golden Path" can get in your way. Rails is very Opinionated Software and that's how Rails works. As long as you follow that, you should be very productivity. If you decide to go off the Rails (i.e. namespaces), it can be difficult.

Rails uses a DSL in its models (i.e. has_many, has_one for relationships) and in the Rails router. It allows you to very simply map a URL to a controller/method. In addition to DSLs, Rails has first-class testing and its generators create stub tests for you.

Bad things about Rails: too much magic, moves to fast (too many releases).

Merb
Merb was originally developed by Ezra Zygmuntowicz to run alongside a Rails app to handle file uploads. It grew from there and became it's own beast. Merb is very much about using only what you need. It has "package repos" that allow you to add additional features. For example, merb-core doesn't contain an ORM framework, just a web framework. Merb also allows you to choose your ORM. It's standardized on Rack, so it can run on just about any web server. It also included "deferred actions" that allow you to send some URLs to evented web servers and others to threaded web servers. Merb eschews the "magic" that Rails has. It tries to stay away from making it's code a "monument to personal cleverness". Simple code scales better and runs faster.

One of the downers to Merb is that it's flexibility allows you to get down to the nitty gritty. However, it can be less productive than Rails because of its flexibility. Another downside is its documentation and examples are sparse. Merb is not recommended if you're just getting into Ruby.

Camping
Camping is a micro framework (< 4K) developed by why the lucky stiff. It's designed to develop small applications. You can do everything in one file and create prototypes very quickly. It uses Markaby to write HTML code in a builder-style fashion.

Since a Camping application is all in one file, it can be difficult to develop large applications. The solution is to write small apps and mount them in the same URL space. The only issue with small apps sharing the same space is they have to use the same database. One downside to Camping is there is no standard test framework. Mosquito was developed as a solution, but doesn't seem to be maintained.

Sinatra
Simple. Fast. Effective. It's designed to allow creating REST applications with minimal dependencies. Similar to Camping, it has one file for the entire application. Unlike Camping, Sinatra doesn't follow MVC conventions, so it may be difficult to port a Sinatra application to another framework.

Posted in Open Source at Jul 23 2008, 12:33:08 PM MDT 1 Comment

[OSCON 2008] The Keynote

This morning, I woke up awful early to polish my presentation, walked to the train station and rode Amtrak from Salem to Portland. The commute was great: there's nothing better than traveling with power and an EVDO card + the option to get a cup of coffee. After getting off the train, riding The Max and walking to the Oregon Convention Center, I'm now sitting in the Keynote at OSCON. Here's my notes from this session.

10 years ago, leaders of the free software movement got together and tried to figure out a way to help people understand how to get access to software freedom. As they talked, there was a gradual meeting of minds. Finally, one person suggested "Open Source". A few weeks later, there was a larger meeting of people and they heard about this term. It was an idea that changed the idea of software freedom and what free software was. We've come along way since then. Last year, we heard about open source and and it trying to find identity in corporations. This year, we're hearing about corporations trying to find their identity in open source.

The official tag for this conference is: oscon08.

Tim O'Reilly
While this is the 10th Anniversary of OSCON, it's also the 12th Anniversary of the O'Reilly Perl Conference (where it all started). Tim began his activism with Perl when it got on the web. He was thinking about the internet and the online world, from the beginning (when many others were coming from Linux). Open Source was almost named "SourceWare". Tim believes his biggest contribution is bringing Open Source and the Internet together.

"Keep your History" - make the things you put online accessible for years to come.

When OSCON first started, it was all about the OS Wars. Tim is showing a shirt with the famous Ghandi quote on it about "First they laugh at you..." and it has a Linux logo on the bottom. It's seems ironic that Microsoft is now one of the major sponsors of this conference (my thoughts, not Tim's).

Open Source Technology in the Enterprise. IT jobs are 2.3% of all jobs posted, according to the Bureau of Labor Statistics. Technology oriented companies (e.g., Google, Yahoo, Sun) make heavy use of Open Source (40% of all jobs posted by Y!). Open source is growing faster in non-tech companies. Of the open source technologies in the enterprise, the highest share of jobs is Linux (19%), followed by Perl, JavaScript and PHP. As far as the fastest growing, Django and Alfresco are at the top.

Three Big Challenges and Opportunities:

  1. Cloud Computing
  2. The (Open) Programmable Web
  3. Open Mobile

Cloud Computing: Amazon Web Services, Google App Engine, The Engine Yard, etc.

Jesse Vincent: "Web 2.0 is Sharecropping"

Danny O'Brien: "If we want people to have the same degree of user autonomy as we’ve come to expect from the world, we may have to sit down and code alternatives to Google Docs, Twitter, and EC3 that can live with us oon the edge, not be run by third parties."

Basically, Tim is saying the that cloud computing is great, but it doesn't fit well with open source. This is primarily because if you build on a cloud, you have to be careful not to get locked into that platform.

Data is the "Intel Inside".

The Web is the Internet Operating System - the subsystems will be data subsystems.

Locking in data: iTunes and Amazon's Kindle. On the other hand you have Yahoo's BOSS, which is doing the opposite.

We Need the Open Web Platform! Tomorrow's Keynote, "Supporting the Open Web" will talk much more about this.

The Mobile Web has caused the "browser wars" to resurface. However, big companies like Google are putting a stake in the ground and saying "We believe in open". Net Neutrality and The Open Handset Alliance are two of Google's smartest strategic decisions. They understand how much they depend on the open ecosystem.

When we look at our success in the last 10 years, we can be really excited. But what's really impressive is how much we (as an open source community) is how we've risen to new challenges and challenged the openness of new platforms and industries.

Christine Peterson
Christine is the President of the the Foresight Institute. Christine was the person who suggested the term "open source" at the meeting referenced above. Unfortunately, my first battery died as Christine was coming on stage, so I missed writing down the first 10 minutes of her 15 minute talk. She's talking about the openness vs. privacy of keeping US citizens safe. She started her talk apologizing for the ethnocentricity of her talk and moved to quickly note that the e-voting controversy wouldn't have happened if open source software was used.

"Who would have guessed that the folks with the pocket protectors would turn out to be the ones with the right stuff?" -- LA Times

Founding Geeks: Thomas Jefferson (mechanical geek) and Thomas Edison (electricity geek).

You can't just complain about things. The fear is real. We can't just complain about how DC is solving problems, we have to step up and solve them ourselves.

"No Secret Software for Public Sensing Data."

Dirk Hohndel
Dirk is the Chief Linux and Open Source Technologist at Intel. He's talking about Moblin: Linux for Next Generation Mobile Internet. This sounds like something that has been talked about a million times before. Why is it interesting today? Because we're at an open source conference and open source is what makes it interesting.

When people look at Intel, they don't think of open source. However, Intel is very involved in open source and uses an open source methodology internally for their development process. They also have one of the largest grids powered by open source (~100K Linux servers).

Moblin is about the internet, about mobility, about flexibility and extensibility. What's happening today is the ideas of 10 years ago have become affordable to produce (for manufacturers) and purchase (for consumers). There's lots of proprietary ways to develop the mobile web, but it needs to be open in order to prevent lock-in (to a platform) and encourage innovation.

A year ago, Intel started Moblin. Initially, there wasn't a lot of interest from open source developers. The majority of interest came from companies, particularly hardware vendors. To Dirk, this was disappointing as he really wanted a community to guide the project and make choices about the platform. There's lots of Open Mobile/Linux efforts out there, but there aren't any that are truly open - with access to the source code and everything else you'd expect from an open source project. Intel was hoping to announce a cutting-edge infrastructure for Moblin here at OSCON, but they're a few weeks behind. They hope to be ready for soon.

"The hope that I have is the community takes this from us. Show us where to go. Show us where not to go. Help us get this right."

Tim O'Reilly interviews Monty Widenius and Brian Aker
Tim asks how it's going 6 months in. Monty responds that he's very happy they didn't have to go public and that Sun is still trying to figure out what they bought. One of the things difficulties they've seen about encouraging Sun's engineers to be involved in open source is some are hesitant about open sourcing their code. The biggest problem is engineers are afraid of the feedback/scrutiny that their code will receive.

MySQL was very unique as a company in that it was a virtual company, with most engineers working out of their homes. MySQL has become an enabling force for moving Sun to a similar model.

Monty is working on Maria (new storage engine) and Brian is working on Drizzle (a slimmer version of MySQL). Drizzle was inspired by a conversation when Brian was talking to Rackspace's CTO.

"Do less and then create extensibility mechanisms." -- Tim O'Reilly

Posted in Open Source at Jul 23 2008, 11:29:21 AM MDT Add a Comment