Raible Designs

Building a REST API with JAXB, Spring Boot and Spring Data

2014-10-29T05:52:37-06:00

If someone asked you to develop a REST API on the JVM, which frameworks would you use? I was recently tasked with such a project. My client asked me to implement a REST API to ingest requests from a 3rd party. The project entailed consuming XML requests, storing the data in a database, then exposing the data to internal application with a JSON endpoint. Finally, it would allow taking in a JSON request and turning it into an XML request back to the 3rd party.

With the recent release of Apache Camel 2.14 and my success using it, I started by copying my Apache Camel / CXF / Spring Boot project and trimming it down to the bare essentials. I whipped together a simple Hello World service using Camel and Spring MVC. I also integrated Swagger into both. Both implementations were pretty easy to create (sample code), but I decided to use Spring MVC. My reasons were simple: its REST support was more mature, I knew it well, and Spring MVC Test makes it easy to test APIs.

Camel's Swagger support without web.xml
As part of the aforementioned spike, I learned out how to configure Camel's REST and Swagger support using Spring's JavaConfig and no web.xml. I made this into a sample project and put it on GitHub as camel-rest-swagger.

This article shows how I built a REST API with Java 8, Spring Boot/MVC, JAXB and Spring Data (JPA and REST components). I stumbled a few times while developing this project, but figured out how to get over all the hurdles. I hope this helps the team that's now maintaining this project (my last day was Friday) and those that are trying to do something similar.

XML to Java with JAXB

The data we needed to ingest from a 3rd party was based on the NCPDP Standards. As a member, we were able to download a number of XSD files, put them in our project and generate Java classes to handle the incoming/outgoing requests. I used the maven-jaxb2-plugin to generate the Java classes.

<plugin>
    <groupId>org.jvnet.jaxb2.maven2</groupId>
    <artifactId>maven-jaxb2-plugin</artifactId>
    <version>0.8.3</version>
    <executions>
        <execution>
            <goals>
                <goal>generate</goal>
            </goals>
            <configuration>
                <args>
                    <arg>-XtoString</arg>
                    <arg>-Xequals</arg>
                    <arg>-XhashCode</arg>
                    <arg>-Xcopyable</arg>
                </args>
                <plugins>
                    <plugin>
                        <groupId>org.jvnet.jaxb2_commons</groupId>
                        <artifactId>jaxb2-basics</artifactId>
                        <version>0.6.4</version>
                    </plugin>
                </plugins>
                <schemaDirectory>src/main/resources/schemas/ncpdp</schemaDirectory>
            </configuration>
        </execution>
    </executions>
</plugin>

The first error I ran into was about a property already being defined.

[INFO] --- maven-jaxb2-plugin:0.8.3:generate (default) @ spring-app ---
[ERROR] Error while parsing schema(s).Location [ file:/Users/mraible/dev/spring-app/src/main/resources/schemas/ncpdp/structures.xsd{1811,48}].
com.sun.istack.SAXParseException2; systemId: file:/Users/mraible/dev/spring-app/src/main/resources/schemas/ncpdp/structures.xsd;
    lineNumber: 1811; columnNumber: 48; Property "MultipleTimingModifierAndTimingAndDuration" is already defined.
    Use <jaxb:property> to resolve this conflict.
at com.sun.tools.xjc.ErrorReceiver.error(ErrorReceiver.java:86)

I was able to workaround this by upgrading to maven-jaxb2-plugin version 0.9.1. I created a controller and stubbed out a response with hard-coded data. I confirmed the incoming XML-to-Java marshalling worked by testing with a sample request provided by our 3rd party customer. I started with a curl command, because it was easy to use and could be run by anyone with the file and curl installed.

curl -X POST -H 'Accept: application/xml' -H 'Content-type: application/xml' \
--data-binary @sample-request.xml http://localhost:8080/api/message -v

This is when I ran into another stumbling block: the response wasn't getting marshalled back to XML correctly. After some research, I found out this was caused by the lack of @XmlRootElement annotations on my generated classes. I posted a question to Stack Overflow titled Returning JAXB-generated elements from Spring Boot Controller. After banging my head against the wall for a couple days, I figured out the solution.

I created a bindings.xjb file in the same directory as my schemas. This causes JAXB to generate @XmlRootElement on classes.

<?xml version="1.0"?>
<jxb:bindings version="1.0"
              xmlns:xsd="http://www.w3.org/2001/XMLSchema"
              xmlns:jxb="http://java.sun.com/xml/ns/jaxb"
              xmlns:xjc="http://java.sun.com/xml/ns/jaxb/xjc"
              xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
              xsi:schemaLocation="http://java.sun.com/xml/ns/jaxb http://java.sun.com/xml/ns/jaxb/bindingschema_2_0.xsd">

    <jxb:bindings schemaLocation="transport.xsd" node="/xsd:schema">
        <jxb:globalBindings>
            <xjc:simple/>
        </jxb:globalBindings>
    </jxb:bindings>
</jxb:bindings>

To add namespaces prefixes to the returned XML, I had to modify the maven-jaxb2-plugin to add a couple arguments.

<arg>-extension</arg>
<arg>-Xnamespace-prefix</arg>

And add a dependency:

<dependencies>
    <dependency>
        <groupId>org.jvnet.jaxb2_commons</groupId>
        <artifactId>jaxb2-namespace-prefix</artifactId>
        <version>1.1</version>
    </dependency>
</dependencies>

Then I modified bindings.xjb to include the package and prefix settings. I also moved <xjc:simple/> into a global setting. I eventually had to add prefixes for all schemas and their packages.

<?xml version="1.0"?>
<bindings version="2.0" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns="http://java.sun.com/xml/ns/jaxb"
          xmlns:xjc="http://java.sun.com/xml/ns/jaxb/xjc" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xmlns:namespace="http://jaxb2-commons.dev.java.net/namespace-prefix"
          xsi:schemaLocation="http://java.sun.com/xml/ns/jaxb http://java.sun.com/xml/ns/jaxb/bindingschema_2_0.xsd
              http://jaxb2-commons.dev.java.net/namespace-prefix http://java.net/projects/jaxb2-commons/sources/svn/content/namespace-prefix/trunk/src/main/resources/prefix-namespace-schema.xsd">

    <globalBindings>
        <xjc:simple/>
    </globalBindings>

    <bindings schemaLocation="transport.xsd" node="/xsd:schema">
        <schemaBindings>
            <package name="org.ncpdp.schema.transport"/>
        </schemaBindings>
        <bindings>
            <namespace:prefix name="transport"/>
        </bindings>
    </bindings>
</bindings>

I learned how to add prefixes from the namespace-prefix plugins page.

Finally, I customized the code-generation process to generate Joda Time's DateTime instead of the default XMLGregorianCalendar. This involved a couple custom XmlAdapters and a couple additional lines in bindings.xjb. You can see the adapters and bindings.xjb with all necessary prefixes in this gist. Nicolas Fränkel's Customize your JAXB bindings was a great resource for making all this work.

I wrote a test to prove that the ingest API worked as desired.

@RunWith(SpringJUnit4ClassRunner.class)
@SpringApplicationConfiguration(classes = Application.class)
@WebAppConfiguration
@DirtiesContext(classMode = DirtiesContext.ClassMode.AFTER_CLASS)
public class InitiateRequestControllerTest {

    @Inject
    private InitiateRequestController controller;

    private MockMvc mockMvc;

    @Before
    public void setup() {
        MockitoAnnotations.initMocks(this);
        this.mockMvc = MockMvcBuilders.standaloneSetup(controller).build();
    }

    @Test
    public void testGetNotAllowedOnMessagesAPI() throws Exception {
        mockMvc.perform(get("/api/initiate")
                .accept(MediaType.APPLICATION_XML))
                .andExpect(status().isMethodNotAllowed());
    }

    @Test
    public void testPostPaInitiationRequest() throws Exception {
        String request = new Scanner(new ClassPathResource("sample-request.xml").getFile()).useDelimiter("\\Z").next();

        mockMvc.perform(post("/api/initiate")
                .accept(MediaType.APPLICATION_XML)
                .contentType(MediaType.APPLICATION_XML)
                .content(request))
                .andExpect(status().isOk())
                .andExpect(content().contentType(MediaType.APPLICATION_XML))
                .andExpect(xpath("/Message/Header/To").string("3rdParty"))
                .andExpect(xpath("/Message/Header/SenderSoftware/SenderSoftwareDeveloper").string("HID"))
                .andExpect(xpath("/Message/Body/Status/Code").string("010"));
    }
}

Spring Data for JPA and REST

With JAXB out of the way, I turned to creating an internal API that could be used by another application. Spring Data was fresh in my mind after reading about it last summer. I created classes for entities I wanted to persist, using Lombok's @Data to reduce boilerplate.

I read the Accessing Data with JPA guide, created a couple repositories and wrote some tests to prove they worked. I ran into an issue trying to persist Joda's DateTime and found Jadira provided a solution.

I added its usertype.core as a dependency to my pom.xml:

<dependency>
    <groupId>org.jadira.usertype</groupId>
    <artifactId>usertype.core</artifactId>
    <version>3.2.0.GA</version>
</dependency>

... and annotated DateTime variables accordingly.

@Column(name = "last_modified", nullable = false)
@Type(type="org.jadira.usertype.dateandtime.joda.PersistentDateTime")
private DateTime lastModified;

With JPA working, I turned to exposing REST endpoints. I used Accessing JPA Data with REST as a guide and was looking at JSON in my browser in a matter of minutes. I was surprised to see a "profile" service listed next to mine, and posted a question to the Spring Boot team. Oliver Gierke provided an excellent answer.

Swagger

Spring MVC's integration for Swagger has greatly improved since I last wrote about it. Now you can enable it with a @EnableSwagger annotation. Below is the SwaggerConfig class I used to configure Swagger and read properties from application.yml.

@Configuration
@EnableSwagger
public class SwaggerConfig implements EnvironmentAware {
	public static final String DEFAULT_INCLUDE_PATTERN = "/api/.*";

	private RelaxedPropertyResolver propertyResolver;

	@Override
	public void setEnvironment(Environment environment) {
		this.propertyResolver = new RelaxedPropertyResolver(environment, "swagger.");
	}

	/**
	 * Swagger Spring MVC configuration
	 */
	@Bean
	public SwaggerSpringMvcPlugin swaggerSpringMvcPlugin(SpringSwaggerConfig springSwaggerConfig) {
		return new SwaggerSpringMvcPlugin(springSwaggerConfig)
				.apiInfo(apiInfo())
				.genericModelSubstitutes(ResponseEntity.class)
				.includePatterns(DEFAULT_INCLUDE_PATTERN);
	}

	/**
	 * API Info as it appears on the swagger-ui page
	 */
	private ApiInfo apiInfo() {
		return new ApiInfo(
				propertyResolver.getProperty("title"),
				propertyResolver.getProperty("description"),
				propertyResolver.getProperty("termsOfServiceUrl"),
				propertyResolver.getProperty("contact"),
				propertyResolver.getProperty("license"),
				propertyResolver.getProperty("licenseUrl"));
	}
}

After getting Swagger to work, I discovered that endpoints published with @RepositoryRestResource aren't picked up by Swagger. There is an open issue for Spring Data support in the swagger-springmvc project.

Liquibase Integration

I configured this project to use H2 in development and PostgreSQL in production. I used Spring profiles to do this and copied XML/YAML (for Maven and application*.yml files) from a previously created JHipster project.

Next, I needed to create a database. I decided to use Liquibase to create tables, rather than Hibernate's schema-export. I chose Liquibase over Flyway based of discussions in the JHipster project. To use Liquibase with Spring Boot is dead simple: add the following dependency to pom.xml, then place changelog files in src/main/resources/db/changelog.

<dependency>
    <groupId>org.liquibase</groupId>
    <artifactId>liquibase-core</artifactId>
</dependency>

I started by using Hibernate's schema-export and changing hibernate.ddl-auto to "create-drop" in application-dev.yml. I also commented out the liquibase-core dependency. Then I setup a PostgreSQL database and started the app with "mvn spring-boot:run -Pprod".

I generated the liquibase changelog from an existing schema using the following command (after downloading and installing Liquibase).

liquibase --driver=org.postgresql.Driver --classpath="/Users/mraible/.m2/repository/org/postgresql/postgresql/9.3-1102-jdbc41/postgresql-9.3-1102-jdbc41.jar:/Users/mraible/snakeyaml-1.11.jar" --changeLogFile=/Users/mraible/dev/spring-app/src/main/resources/db/changelog/db.changelog-02.yaml --url="jdbc:postgresql://localhost:5432/mydb" --username=user --password=pass generateChangeLog

I did find one bug - the generateChangeLog command generates too many constraints in version 3.2.2. I was able to fix this by manually editing the generated YAML file.

Tip: If you want to drop all tables in your database to verify Liquibase creation is working in PostgeSQL, run the following commands:

psql -d mydb
drop schema public cascade;
create schema public;

After writing minimal code for Spring Data and configuring Liquibase to create tables/relationships, I relaxed a bit, documented how everything worked and added a LoggingFilter. The LoggingFilter was handy for viewing API requests and responses.

@Bean
public FilterRegistrationBean loggingFilter() {
    LoggingFilter filter = new LoggingFilter();
    FilterRegistrationBean registrationBean = new FilterRegistrationBean();
    registrationBean.setFilter(filter);
    registrationBean.setUrlPatterns(Arrays.asList("/api/*"));
    return registrationBean;
}

Accessing API with RestTemplate

The final step I needed to do was figure out how to access my new and fancy API with RestTemplate. At first, I thought it would be easy. Then I realized that Spring Data produces a HAL-compliant API, so its content is embedded inside an "_embedded" JSON key.

After much trial and error, I discovered I needed to create a RestTemplate with HAL and Joda-Time awareness.

@Bean
public RestTemplate restTemplate() {
	ObjectMapper mapper = new ObjectMapper();
	mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
	mapper.registerModule(new Jackson2HalModule());
	mapper.registerModule(new JodaModule());

	MappingJackson2HttpMessageConverter converter = new MappingJackson2HttpMessageConverter();
	converter.setSupportedMediaTypes(MediaType.parseMediaTypes("application/hal+json"));
	converter.setObjectMapper(mapper);
	StringHttpMessageConverter stringConverter = new StringHttpMessageConverter();
	stringConverter.setSupportedMediaTypes(MediaType.parseMediaTypes("application/xml"));

	List<HttpMessageConverter<?>> converters = new ArrayList<>();
	converters.add(converter);
	converters.add(stringConverter);

	return new RestTemplate(converters);
}

The JodaModule was provided by the following dependency:

<dependency>
    <groupId>com.fasterxml.jackson.datatype</groupId>
    <artifactId>jackson-datatype-joda</artifactId>
</dependency>

With the configuration complete, I was able to write a MessagesApiITest integration test that posts a request and retrieves it using the API. The API was secured using basic authentication, so it took me a bit to figure out how to make that work with RestTemplate. Willie Wheeler's Basic Authentication With Spring RestTemplate was a big help.

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(classes = IntegrationTestConfig.class)
public class MessagesApiITest {
    private final static Log log = LogFactory.getLog(MessagesApiITest.class);
    @Value("http://${app.host}/api/initiate")
    private String initiateAPI;
    @Value("http://${app.host}/api/messages")
    private String messagesAPI;
    @Value("${app.host}")
    private String host;
    @Inject
    private RestTemplate restTemplate;

    @Before
    public void setup() throws Exception {
        String request = new Scanner(new ClassPathResource("sample-request.xml").getFile()).useDelimiter("\\Z").next();

        ResponseEntity<org.ncpdp.schema.transport.Message> response = restTemplate.exchange(getTestUrl(initiateAPI),
                HttpMethod.POST, getBasicAuthHeaders(request), org.ncpdp.schema.transport.Message.class,
                Collections.emptyMap());
        assertEquals(HttpStatus.OK, response.getStatusCode());
    }

    @Test
    public void testGetMessages() {
        HttpEntity<String> request = getBasicAuthHeaders(null);

        ResponseEntity<PagedResources<Message>> result = restTemplate.exchange(getTestUrl(messagesAPI), HttpMethod.GET,
                request, new ParameterizedTypeReference<PagedResources<Message>>() {});
        HttpStatus status = result.getStatusCode();
        Collection<Message> messages = result.getBody().getContent();

        log.debug("messages found: " + messages.size());
        assertEquals(HttpStatus.OK, status);
        for (Message message : messages) {
            log.debug("message.id: " + message.getId());
            log.debug("message.dateCreated: " + message.getDateCreated());
        }
    }

    private HttpEntity<String> getBasicAuthHeaders(String body) {
        String plainCreds = "user:pass";
        byte[] plainCredsBytes = plainCreds.getBytes();
        byte[] base64CredsBytes = Base64.encodeBase64(plainCredsBytes);
        String base64Creds = new String(base64CredsBytes);

        HttpHeaders headers = new HttpHeaders();
        headers.add("Authorization", "Basic " + base64Creds);
        headers.add("Content-type", "application/xml");

        if (body == null) {
            return new HttpEntity<>(headers);
        } else {
            return new HttpEntity<>(body, headers);
        }
    }
}

To get Spring Data to populate the message id, I created a custom RestConfig class to expose it. I learned how to do this from Tommy Ziegler.

/**
 * Used to expose ids for resources.
 */
@Configuration
public class RestConfig extends RepositoryRestMvcConfiguration {

    @Override
    protected void configureRepositoryRestConfiguration(RepositoryRestConfiguration config) {
        config.exposeIdsFor(Message.class);
        config.setBaseUri("/api");
    }
}

Summary

This article explains how I built a REST API using JAXB, Spring Boot, Spring Data and Liquibase. It was relatively easy to build, but required some tricks to access it with Spring's RestTemplate. Figuring out how to customize JAXB's code generation was also essential to make things work.

I started developing the project with Spring Boot 1.1.7, but upgraded to 1.2.0.M2 after I found it supported Log4j2 and configuring Spring Data REST's base URI in application.yml. When I handed the project off to my client last week, it was using 1.2.0.BUILD-SNAPSHOT because of a bug when running in Tomcat.

This was an enjoyable project to work on. I especially liked how easy Spring Data makes it to expose JPA entities in an API. Spring Boot made things easy to configure once again and Liquibase seems like a nice tool for database migrations.

If someone asked me to develop a REST API on the JVM, which frameworks would I use? Spring Boot, Spring Data, Jackson, Joda-Time, Lombok and Liquibase. These frameworks worked really well for me on this particular project.

Developing with AngularJS - Part III: Services

2013-06-25T07:03:26-06:00

This is the 3rd article in a series on my experience developing with AngularJS. I used AngularJS for several months to create a "My Dashboard" feature for a client and learned a whole bunch of Angular goodness along the way. For previous articles, please see Part I: The Basics and Part II: Dialogs and Data.

Angular offers several ways to interact with data from the server. The easiest way is to use the $resource factory, which lets you interact with RESTful server-side data sources. When we started the My Dashboard project, we were hoping to interact with a REST API, but soon found out that it didn't have all the data we needed. Rather than loading the page and then making another request to get its data, we decided to embed the JSON in the page. For communication back to the server, we used our tried-and-true Ajax solution: DWR.

In Angular-speak, services are singletons that carry out specific tasks common to web apps. In other words, they're any $name object that can be injected into a controller or directive. However, as a Java Developer, I tend to think of services as objects that communicate with the server. Angular's documentation on Creating Services shows you various options for registering services. I used the angular.Module api method.

When I last worked on the project, there were only two services in My Dashboard: Widget and Preferences.

The Widget service is used to retrieve the visible widgets for the user. It has two functions that are exposed to controllers: getUserWidgets(type) and getHiddenWidgets(type). The former function is used at the top of WidgetController, while the latter is used for the configuration dialog mentioned in the previous article.

The code for this service is in services.js. The bulk of the logic is in its filterData() function, where it goes through a 4-step process:

Get all the widgets by type, ensuring they're unique.
Remove the widgets that are hidden by the user's preferences.
Build an array that's ordered by user's preferences.
Add any new widgets that aren't hidden or ordered.

The code for the Widget object is as follows:

angular.module('dashboard.services', []).
    factory('Widget',function ($filter, Preferences) {
        var filter = $filter('filter');
        var unique = $filter('unique');

        function filterData(array, query) {
            // get all possible widgets for a particular type
            var data = filter(array, query);
            data = unique(data);

            // remove widgets that are hidden by users preference
            var hidden = Preferences.getHiddenWidgets(query.type);
            for (var i = 0; i < hidden.length; i++) {
                var w = filter(data, {id: hidden[i]});
                $.each(w, function (index, item) {
                    var itemId = item.id;
                    if (hidden.indexOf(itemId) > -1) {
                        data.splice(data.indexOf(item), 1);
                    }
                });
            }

            // build an array that's ordered by users preference
            var ordered = [];
            var visible = Preferences.getUserWidgets(query.type);
            for (var j = 0; j < visible.length; j++) {
                var v = filter(data, {id: visible[j]});
                $.each(v, function (index, item) {
                    var itemId = item.id;
                    if (visible.indexOf(itemId) > -1) {
                        ordered.push(item)
                    }
                });
            }

            // loop through data again and add any new widgets not in ordered
            $.each(data, function (index, item) {
                if (ordered.indexOf(item) === -1) {
                    ordered.push(item);
                }
            });

            return ordered;
        }

        return {
            getUserWidgets: function (type) {
                return filterData(widgetData, {type: type})
            },

            getHiddenWidgets: function (type) {
                var hidden = Preferences.getHiddenWidgets(type);
                var widgetsForType = filter(widgetData, {type: type});
                widgetsForType = unique(widgetsForType);
                var widgets = [];
                for (var j = 0; j < hidden.length; j++) {
                    var v = filter(widgetsForType, {id: hidden[j]});
                    $.each(v, function (index, item) {
                        if (widgetsForType.indexOf(item) > -1) {
                            widgets.push(item)
                        }
                    });
                }
                return widgets;
            }
        }
    })

Once you have a service configured like this, you can inject it by name. For example, WidgetController has Widget injected into its constructor:

function WidgetController($dialog, $scope, Widget, Preferences) {

Preferences Service

The Preferences service is used to get and save user preferences. It's pretty straightforward and the bulk of its code is interacting with DWR. This service has 5 methods:

getHiddenWidgets(type) - used by Widget service
getUserWidgets(type) - used by Widget service
saveBarOrder(bars) - called from WidgetController
saveWidgetOrder(type, widgets) - called from WidgetController
saveWidgetPreferences(type, widgets) - called from WidgetController

First, let's take a look at the save*Order() functions. There are two parts of the page that use the ui-sortable directive to initialize drag-and-drop functionality. The first is on the main <ul> that holds the 3 bars on the left.

<ul class="widgets" ui-sortable="{handle:'.heading', update: updateBars}">

The "update" property in the configuration JSON indicates which method to call in the controller. Similarly, the tasks and summary items call an updateOrder function.

<ul class="summary-items" ng-model="summaryWidgets" ui-sortable="{update: updateOrder}">
...
<ul class="task-items" ng-model="taskWidgets" ui-sortable="{update: updateOrder}">

These functions are in WidgetController and build an array of widget ids to pass to the Preferences service.

$scope.updateBars = function(event, ui) {
    var bars = [];
    $.each($(ui.item).parent().children(), function (index, item) {
        bars.push(item.id.substring(0, item.id.indexOf('-')))
    });
    Preferences.saveBarOrder(bars);
};

$scope.updateOrder = function(event, ui) {
    var parentId = $(ui.item).parent().parent().attr('id');
    var type = parentId.substring(0, parentId.indexOf('-'));
    var items = [];
    $.each($(ui.item).parent().children(), function (index, item) {
        items.push(item.id.substring(item.id.indexOf('-') + 1))
    });
    Preferences.saveWidgetOrder(type, {items: items});
};

The bar order is used when the page is loaded. The following scriptlet code exists at the bottom of the app's page, in its $(document).ready:

<%  String barOrder = user.getDashboardBarSortOrder();
    if (barOrder != null) { %>
    sortBars(['<%= barOrder %>']);
<% } %>

The sortBars() function is in a dashboard.js file (where we put all non-Angular functions):

function sortBars(barOrder) {
    // Sort bars according to user preferences
    $.each(barOrder, function(index, item) {
        var bar = $('#' + item + '-bar');
        if (bar.index() !== index) {
            if (index === 0) {
                bar.insertBefore($('.widgets>li:first-child'));
            } else if (index === (barOrder.length - 1)) {
                bar.insertAfter($('.widgets>li:last-child'));
            } else {
                bar.insertBefore($('.widgets>li:eq(' + index + ')'));
            }
        }
    });
}

Now that you've seen where Preferences is called from, let's take a look at the code for the service.

The checks for undefined and uniqueness in the code below shouldn't be necessary, but I prefer defensive coding.

factory('Preferences', function ($filter) {
    var unique = $filter('unique');

    return {
        // Get in-page variable: hiddenWidgets
        getHiddenWidgets: function (type) {
            var items = hiddenWidgets[type];
            return (angular.isUndefined(items) ? [] : unique(items));
        },

        // Get in-page variable: userWidgets
        getUserWidgets: function (type) {
            var items = userWidgets[type];
            return (angular.isUndefined(items) ? [] : unique(items));
        },

        // Save main bar (task, summary, chart) order
        saveBarOrder: function (bars) {
            DWRFacade.saveDashboardBarSortOrder(bars, {
                errorHandler: function (errorString) {
                    alert(errorString);
                }
            })
        },

        // Save order of widgets from sortable
        saveWidgetOrder: function (type, widgets) {
            userWidgets[type] = widgets.items;
            DWRFacade.saveDashboardWidgetPreference(type, widgets, {
                errorHandler: function (errorString) {
                    alert(errorString);
                }
            });
        },

        // Save hidden and visible (and order) widgets from config dialog
        saveWidgetPreferences: function (type, widgets) {
            // widgets is a map of hidden and visible
            var hiddenIds = [];
            $.each(widgets.hidden, function (index, item) {
                hiddenIds.push(item.id);
            });
            var visibleIds = [];
            $.each(widgets.items, function (index, item) {
                visibleIds.push(item.id);
            });
            var preferences = {
                hidden: hiddenIds,
                items: visibleIds
            };
            // reset local variables in page
            hiddenWidgets[type] = hiddenIds;
            userWidgets[type] = visibleIds;
            DWRFacade.saveDashboardWidgetPreference(type, preferences, {
                errorHandler: function (errorString) {
                    alert(errorString);
                }
            });
        }
    }
})

Using $http and Receiving Data

In this particular application, we didn't do any reading from the server with Angular. We simply wrote preferences to the server, and updated embedded variables when data changed. Real-time functionality of the app wouldn't be noticeable if a write failed.

In my current Angular project, it's more of a full-blown application that does as much reading as writing. For this, I've found it useful to either 1) pass in callbacks to services or 2) use Angular's event system to publish/subscribe to events.

The first method is the easiest, and likely the most familiar to JavaScript developers. For example, here's the controller code to remove a profile picture:

Profile.removePhoto($scope.user, function (data) {
    // close the dialog
    $scope.close('avatar');
    // success message using toastr: http://bit.ly/14Uisgm
    Flash.pop({type: 'success', body: 'Your profile picture was removed.'});
})

And the Profile.removePhoto() method:

removePhoto: function (user, callback) {
    $http.post('/profile/removePhoto', user).success(function (response) {
        return callback(response);
    });
}

The second, event-driven method works equally as well, but can easily suffer from typos in event names.

// controller calling code
Profile.getUser();

// service code
getUser: function () {
    $http.get('/profile').success(function (data) {
        if (data.username) {
            $log.info('Profile for ' + data.username + ' retrieved!');
            $rootScope.$broadcast('event:profile', data);
        }
    });
}

// controller receiving code
$rootScope.$on('event:profile', function (event, data) {
    $scope.user = data;
});

I like both methods, but the event-driven one seems like it could offer more extensibility in the future.

Summary

Using in-page variables and DWR doesn't seem to be recommended by the Angular Team. However, it worked well for us and seems like a good way to construct Angular services. Even if a REST API becomes available to get all the data, I think using in-page variables to minimize requests is a good idea.

When retrieving data, you can use callbacks or Angular's pub/sub event system ($broadcast and $on) to get data in your controllers. If you want to learn more about this technique, see Eric Terpstra's Communicating with $broadcast. In his article, Eric mentions Thomas Burleson's pub/sub module that acts as a message queue. If you've used Thomas's MessageQueue (or something similar) with Angular, I'd love to hear about your experience.

In the next article, I'll talk about how we redesigned My Dashboard and used CSS3 and JavaScript to implement new ideas.

Secure JSON Services with Play Scala and SecureSocial

2012-02-12T16:02:43-07:00

Last November, I traveled to Antwerp to speak at Devoxx. After my talk on HTML5 with Play Scala, Mattias Karlsson approached me and we had a chat about doing the same talk at Jfokus in Stockholm. I agreed and we began talking details after Trish and I returned to the US.

I wrote this article on a plane between Denver and Seattle and will be hopping over the North Pole to Stockholm via Iceland tonight. For the past couple of weeks, I've been updating my Play More! HTML5/mobile app to add some new features. Most notably, I wanted to upgrade to Play 2.0, create JSON services and add authentication.

Upgrading to Play 2.0
My attempt to upgrade to Play 2.0 involved checking out the source from GitHub, building and installing the RC1 snapshot. As I tried to upgrade my app and started getting failed imports, I turned to the internet (specifically StackOverflow) to see if it was a good idea. The first answer for that question suggested I stay with 1.x.

If it's a critical project, to be finished before next March 2012, I would go with Play 1.x. If it's a less important project, which could be delayed, and that in any case won't be released before March 2012, try Play 2.0.

While I didn't plan on releasing Play More! before Jfokus, I decided upgrading didn't add a whole lot to the talk. Also, I couldn't find a Play Scala 0.9.1 to Play 2.0 upgrade guide and I didn't have enough time to create one. So I decided to stick with Play 1.2.4 and add some JSON services for my iPhone client.

JSON Servers
I found Manuel Bernhardt's Play! Scala and JSON. This led me to Jerkson, built by the now infamous @coda. I was able to easily get things working fairly quickly and wrote the following WorkoutService.scala:

package controllers.api

import play.mvc.Controller
import models._
import com.codahale.jerkson.Json._

object WorkoutService extends Controller {

  def workouts = {
    response.setContentTypeIfNotSet("application/json")
    generate(Workout.find().list())
  }
  def edit(id: Long) = {
    generate(Workout.byIdWithAthleteAndComments(id))
  }

  def create() = {
    var workout = params.get("workout", classOf[Workout])
    Workout.create(workout)
  }

  def save(id: Option[Long]) = {
    var workout = params.get("workout", classOf[Workout])
    Workout.update(workout)
  }

  def delete(id: Long) = {
    Workout.delete("id={id}").on("id" -> id).executeUpdate()
  }
}

Next, I added routes for my new API to conf/routes:

GET     /api/workouts               api.WorkoutService.workouts
GET     /api/workout/{id}           api.WorkoutService.edit
POST    /api/workout                api.WorkoutService.create
PUT     /api/workout/{id}           api.WorkoutService.save
DELETE  /api/workout/{id}           api.WorkoutService.delete

Then I created an ApiTest.scala class that verifies the first method works as expected.

import play.test.FunctionalTest
import play.test.FunctionalTest._
import org.junit._

class ApiTests extends FunctionalTest {
  
    @Test
    def testGetWorkouts() {
        var response = GET("/api/workouts");
        assertStatus(200, response);
        assertContentType("application/json", response)
        println(response.out)
    }
}

I ran "play test", opened my browser to http://localhost:9000/@tests and clicked ApiTests -> Start to verify it worked. All the green made me happy.

Finally, I wrote some CoffeeScript and jQuery to allow users to delete workouts and make sure delete functionality worked.

$('#delete').click ->
  $.ajax
    type: 'POST'
    url: $(this).attr('rel')
    error: ->
      alert('Delete failed, please try again.')
    success: (data) ->
      location.href = "/more"

I was very impressed with how easy Play made it to create JSON services and I smiled as my CoffeeScript skills got a refresher.

The Friday before we left for Devoxx, I saw the module registration request for SecureSocial.

SecureSocial with Play Scala
From SecureSocial's README:

SecureSocial allows you to add an authentication UI to your app that works with services based on OAuth1, OAuth2, OpenID and OpenID+OAuth hybrid protocols.

It also provides a Username and Password mechanism for users that do not wish to use existing accounts in other networks.

The following services are supported in this release:

Twitter (OAuth1)
Facebook (OAuth2)
Google (OpenID + OAuth Hybrid)
Yahoo (OpenID + OAuth Hybrid)
LinkedIn (OAuth1)
Foursquare (OAuth2)
MyOpenID (OpenID)
Wordpress (OpenID)
Username and Password

In other words, it sounded like a dream come true and I resolved to try it once I found the time. That time found me last Monday evening and I sent a direct message to @jaliss (the module's author) via Twitter.

Does Secure Social work with Play Scala? I'd like to use it in my Play More! project.

Jorge responded 16 minutes later saying that he hadn't used Play Scala and he'd need to do some research. At 8 o'clock that night (1.5 hours after my original DM), Jorge had a sample working and emailed it to me. 10 minutes later I was adding a Secure trait to my project.

package controllers

import play.mvc._
import controllers.securesocial.SecureSocial

/*
 * @author Jorge Aliss <jaliss@gmail.com> of Secure Social fame.
 */
trait Secure {
  self: Controller =>

  @Before def checkAccess() {
    SecureSocial.DeadboltHelper.beforeRoleCheck()
  }

  def currentUser = {
    SecureSocial.getCurrentUser
  }
}

I configured Twitter and Username + Password as my providers by adding the following to conf/application.conf.

securesocial.providers=twitter,userpass

I also had to configure a number of securesocial.twitter.* properties. Next, I made sure my routes were aware of SecureSocial by adding the following to the top of conf/routes:

  *       /auth               module:securesocial

Then I specified it as a dependency in conf/dependencies.yml and ran "play deps".

    - play -> securesocial 0.2.4

After adding "with Secure" to my Profile.scala controller, I tried to access its route and was prompted to login. Right off the bat, I was shown an error about a missing jQuery 1.5.2 file in my "javascripts" folder, so I added it and rejoiced when I was presented with a login screen. I had to add the app on Twitter to use its OAuth servers, but I was pumped when both username/password authentication worked (complete with signup!) as well as Twitter.

The only issue I ran into with SecureSocial was that it didn't find the default implementation of SecureSocial's UserService.Service when running in prod mode. I was able to workaround this by adding a SecureService.scala implementation to my project and coding it to talk to my Athlete model. I didn't bother to hook in creating a new user when they logged in from Twitter, but that's something I'll want to do in the future. I was also pleased to find out customizing SecureSocial's views was a breeze. I simply copied them from the module into my app's views and voila!

package services

import play.db.anorm.NotAssigned
import play.libs.Codec
import collection.mutable.{SynchronizedMap, HashMap}
import models.Athlete
import securesocial.provider.{ProviderType, UserService, SocialUser, UserId}

class SecureService extends UserService.Service {
  val activations = new HashMap[String, SocialUser] with SynchronizedMap[String, SocialUser]

  def find(userId: UserId): SocialUser = {
    val user = Athlete.find("email={email}").on("email" -> userId.id).first()

    user match {
      case Some(user) => {
        val socialUser = new SocialUser
        socialUser.id = userId
        socialUser.displayName = user.firstName
        socialUser.email = user.email
        socialUser.isEmailVerified = true
        socialUser.password = user.password
        socialUser
      }
      case None => {
        if (!userId.provider.eq(ProviderType.userpass)) {
          var socialUser = new SocialUser
          socialUser.id = userId
          socialUser
        } else {
          null
        }
      }
    }
  }

  def save(user: SocialUser) {
    if (find(user.id) == null) {
      val firstName = user.displayName
      val lastName = user.displayName
      Athlete.create(Athlete(NotAssigned, user.email, user.password, firstName, lastName))
    }
  }

  def createActivation(user: SocialUser): String = {
    val uuid: String = Codec.UUID()
    activations.put(uuid, user)
    uuid
  }

  def activate(uuid: String): Boolean = {
    val user: SocialUser = activations.get(uuid).asInstanceOf[SocialUser]
    var result = false

    if (user != null) {
      user.isEmailVerified = true
      save(user)
      activations.remove(uuid)
      result = true
    }

    result
  }

  def deletePendingActivations() {
    activations.clear()
  }
}

Jorge was a great help in getting my authentication needs met and he even wrote a BasicAuth.scala trait to implement Basic Authentication on my JSON services.

package controllers

import _root_.securesocial.provider.{UserService, ProviderType, UserId}
import play._
import play.mvc._
import play.libs.Crypto

import controllers.securesocial.SecureSocial

/*
 * @author Jorge Aliss <jaliss@gmail.com> of Secure Social fame.
 */
trait BasicAuth {
  self: Controller =>

  @Before def checkAccess = {
    if (currentUser != null) {
      // this allows SecureSocial.getCurrentUser() to work.
      renderArgs.put("user", currentUser)
      Continue
    }

    val realm =
      Play.configuration.getProperty("securesocial.basicAuth.realm", "Unauthorized")

    if (request.user == null || request.password == null) {
      Unauthorized(realm)
    } else {
      val userId = new UserId
      userId.id = request.user
      userId.provider = ProviderType.userpass
      val user = UserService.find(userId)

      if (user == null ||
        !Crypto.passwordHash(request.password).equals(user.password)) {
        Unauthorized(realm)
      } else {
        // this allows SecureSocial.getCurrentUser() to work.
        renderArgs.put("user", user)
        Continue
      }
    }
  }

  def currentUser = {
    SecureSocial.getCurrentUser()
  }
}

Summary
My latest pass at developing with Scala and leveraging Play to build my app was a lot of fun. While there were issues with class reloading every-so-often and Scala versions with Scalate, I was able to add the features I wanted. I wasn't able to upgrade to Play 2.0, but I didn't try that hard and figured it's best to wait until its upgrade guide has been published.

I'm excited to describe my latest experience to the developers at Jfokus this week. In addition, the conference has talks on Play 2.0, CoffeeScript, HTML5, Scala and Scalate. I hope to attend many of these and learn some new tricks to improve my skills and my app.

Update: The Delving developers have written an article on Migration to Play 2. While it doesn't provide specific details on what they needed to change, it does have good information on how long it took and things to watch for.

Integrating OAuth with AppFuse and its REST API

2011-07-05T10:56:48-06:00

One of the new features in AppFuse 2.1 is an appfuse-ws archetype. This archetype leverages Enunciate and CXF to create a project with a REST API and generated HTML documentation. Enunciate is a very useful tool, allowing you to develop web services with JAX-RS and JAX-WS annotations and have all types of client libraries generated. For me, it seems very useful for developing the backend of SOFEA (a.k.a. modern) applications.

Back in March, Ryan Heaton published a nice article on Securing Web Services in an Enunciate application. I decided to take his tutorial a step further and not only secure my web services, but also to integrate with OAuth 2. In this tutorial, I'll show you how to create a new application with AppFuse WS, secure it, add OAuth support, and then use a client app to authenticate and retrieve data.

Create a New AppFuse WS Project
Integrate Spring Security and OAuth
Authenticate and Retrieve Data with Client

Create a New AppFuse WS Project
To begin, I visited the Create AppFuse Archetypes page and created a new application using the "Web Services Only" option in the Web Framework dropdown. Below is the command I used to create the "appfuse-oauth" project.

mvn archetype:generate -B -DarchetypeGroupId=org.appfuse.archetypes \
-DarchetypeArtifactId=appfuse-ws-archetype -DarchetypeVersion=2.1.0 \
-DgroupId=org.appfuse.example -DartifactId=appfuse-oauth

After doing this, I started the app using mvn jetty:run and confirmed it started OK. At this point, I was able to view the generated documentation for the application at http://localhost:8080. The screenshot below shows what the app looks like at this point.

NOTE: You might notice the REST endpoint of /{username}. This is a bug in AppFuse 2.1.0 and has been fixed in SVN. It does not affect this tutorial.

Integrate Spring Security and OAuth
I originally tried to integrate Spring Security with Enunciate's Securing Web Services Tutorial. However, it only secures endpoints and doesn't do enough filtering for OAuth support, so I ended up using a custom web.xml. I put this file in src/main/resources and loaded it in my enunciate.xml file. I also upgraded Spring Security and imported my security.xml file.

  <?xml version="1.0"?>
  <enunciate xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xsi:noNamespaceSchemaLocation="http://enunciate.codehaus.org/schemas/enunciate-1.22.xsd">
      ...
      <webapp mergeWebXML="src/main/resources/web.xml"/>
      <modules>
      ...
          <spring-app disabled="false" springVersion="3.0.5.RELEASE">
              <springImport uri="classpath:/applicationContext-resources.xml"/>
              <springImport uri="classpath:/applicationContext-dao.xml"/>
              <springImport uri="classpath:/applicationContext-service.xml"/>
              <springImport uri="classpath:/applicationContext.xml"/>
              <springImport uri="classpath:/security.xml"/>
          </spring-app>
      </modules>
  </enunciate>

Then I created src/main/resources/web.xml with a filter for Spring Security and a DispatcherServlet for OAuth support.

<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns="http://java.sun.com/xml/ns/javaee"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"
         version="3.0">

    <filter>
        <filter-name>securityFilter</filter-name>
        <filter-class>org.springframework.web.filter.DelegatingFilterProxy</filter-class>
        <init-param>
            <param-name>targetBeanName</param-name>
            <param-value>springSecurityFilterChain</param-value>
        </init-param>
    </filter>

    <filter-mapping>
        <filter-name>securityFilter</filter-name>
        <url-pattern>/*</url-pattern>
    </filter-mapping>

    <servlet>
        <servlet-name>appfuse-oauth</servlet-name>
        <servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>
        <load-on-startup>1</load-on-startup>
    </servlet>

    <servlet-mapping>
        <servlet-name>appfuse-oauth</servlet-name>
        <url-pattern>/oauth/*</url-pattern>
    </servlet-mapping>
</web-app>

Next, I created a src/main/resources/security.xml and used it to secure my API, specify a login page, supply the users and integrate OAuth (see the last 4 beans below).

<?xml version="1.0" encoding="UTF-8"?>
<beans:beans xmlns="http://www.springframework.org/schema/security"
             xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xmlns:beans="http://www.springframework.org/schema/beans"
             xmlns:oauth="http://www.springframework.org/schema/security/oauth2"
             xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
                           http://www.springframework.org/schema/security http://www.springframework.org/schema/security/spring-security.xsd
                           http://www.springframework.org/schema/security/oauth2 http://www.springframework.org/schema/security/spring-security-oauth2.xsd">

    <http auto-config="true">
        <intercept-url pattern="/api/**" access="ROLE_USER"/>
        <intercept-url pattern="/oauth/**" access="ROLE_USER"/>
        <intercept-url pattern="/**" access="IS_AUTHENTICATED_ANONYMOUSLY"/>
        <form-login login-page="/login.jsp" authentication-failure-url="/login.jsp?error=true"
                    login-processing-url="/j_security_check"/>
    </http>

    <authentication-manager>
        <authentication-provider>
            <user-service>
                <user name="admin" password="admin" authorities="ROLE_USER,ROLE_ADMIN"/>
                <user name="user" password="user" authorities="ROLE_USER"/>
            </user-service>
        </authentication-provider>
    </authentication-manager>

    <!--hook up the spring security filter chain-->
    <beans:alias name="springSecurityFilterChain" alias="securityFilter"/>

    <beans:bean id="tokenServices"
                class="org.springframework.security.oauth2.provider.token.InMemoryOAuth2ProviderTokenServices">
        <beans:property name="supportRefreshToken" value="true"/>
    </beans:bean>

    <oauth:provider client-details-service-ref="clientDetails" token-services-ref="tokenServices">
        <oauth:verification-code user-approval-page="/oauth/confirm_access"/>
    </oauth:provider>

    <oauth:client-details-service id="clientDetails">
        <!--<oauth:client clientId="my-trusted-client" authorizedGrantTypes="password,authorization_code,refresh_token"/>
        <oauth:client clientId="my-trusted-client-with-secret"
                      authorizedGrantTypes="password,authorization_code,refresh_token" secret="somesecret"/>
        <oauth:client clientId="my-less-trusted-client" authorizedGrantTypes="authorization_code"/>-->
        <oauth:client clientId="ajax-login" authorizedGrantTypes="authorization_code"/>
    </oauth:client-details-service>
</beans:beans>

I used the OAuth for Spring Security sample apps to figure this out. In this example, I used authorizedGrantTypes="authorization_code", but you can see from the commented <oauth:client> elements above that there's a few different options. You should also note that the clientId is hard-coded to "ajax-login", signifying I only want to allow a single application to authenticate.

At this point, I'd like to give a shoutout to Ryan Heaton for creating both Enunciate and Spring Security's OAuth support. Nice work Ryan!

At this point, I needed to do a number of additional tasks to finish integrating oauth. The first was to modify the Jetty Plugin's configuration to 1) run on port 9000, 2) load my custom files and 3) allow jetty:run to recognize Enunciate's generated files. Below is the final configuration in my pom.xml.

<plugin>
    <groupId>org.mortbay.jetty</groupId>
    <artifactId>maven-jetty-plugin</artifactId>
    <version>6.1.26</version>
    <configuration>
        <connectors>
            <connector implementation="org.mortbay.jetty.nio.SelectChannelConnector">
                <port>9000</port>
                <maxIdleTime>60000</maxIdleTime>
            </connector>
        </connectors>
        <webAppConfig>
            <baseResource implementation="org.mortbay.resource.ResourceCollection">
                <resourcesAsCSV>
                    ${basedir}/src/main/webapp,
                    ${project.build.directory}/${project.build.finalName}
                </resourcesAsCSV>
            </baseResource>
            <contextPath>/appfuse-oauth</contextPath>
        </webAppConfig>
        <webXml>${project.build.directory}/${project.build.finalName}/WEB-INF/web.xml</webXml>
    </configuration>
</plugin>

Next, I added the necessary OAuth dependencies for Spring Security to my pom.xml. Since the latest release is a milestone release, I had to add Spring's milestone repo too.

<repository>
    <id>spring-milestone</id>
    <url>http://s3.amazonaws.com/maven.springframework.org/milestone</url>
</repository>
...
<dependency>
    <groupId>org.springframework.security</groupId>
    <artifactId>spring-security-taglibs</artifactId>
    <version>${spring.version}</version>
    <exclusions>
        <exclusion>
            <groupId>org.springframework</groupId>
            <artifactId>spring-web</artifactId>
        </exclusion>
        <exclusion>
            <groupId>org.springframework</groupId>
            <artifactId>spring-support</artifactId>
        </exclusion>
    </exclusions>
</dependency>
<dependency>
    <groupId>org.springframework.security.oauth</groupId>
    <artifactId>spring-security-oauth</artifactId>
    <version>1.0.0.M3</version>
    <exclusions>
        <exclusion>
            <groupId>org.springframework</groupId>
            <artifactId>spring-beans</artifactId>
        </exclusion>
        <exclusion>
            <groupId>org.springframework</groupId>
            <artifactId>spring-context</artifactId>
        </exclusion>
        <exclusion>
            <groupId>org.springframework</groupId>
            <artifactId>spring-core</artifactId>
        </exclusion>
    </exclusions>
</dependency>
<dependency>
    <groupId>org.springframework</groupId>
    <artifactId>spring-webmvc</artifactId>
    <version>${spring.version}</version>
</dependency>
<dependency>
    <groupId>javax.servlet</groupId>
    <artifactId>servlet-api</artifactId>
    <version>2.5</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>javax.servlet</groupId>
    <artifactId>jstl</artifactId>
    <version>1.1.2</version>
</dependency>
<dependency>
    <groupId>taglibs</groupId>
    <artifactId>standard</artifactId>
    <version>1.1.2</version>
</dependency>

Since I named my DispatcherServlet "appfuse-oauth" in web.xml, I created a src/main/webapp/WEB-INF/appfuse-oauth-servlet.xml to configure Spring MVC. I had to create the src/main/webapp/WEB-INF directory.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:context="http://www.springframework.org/schema/context"
       xmlns:mvc="http://www.springframework.org/schema/mvc"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
                http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd
                http://www.springframework.org/schema/mvc http://www.springframework.org/schema/mvc/spring-mvc-3.0.xsd">

    <!-- Scans the classpath of this application for @Components to deploy as beans -->
    <context:component-scan base-package="org.appfuse.examples.webapp"/>

    <!-- Configures the @Controller programming model -->
    <mvc:annotation-driven/>

    <!-- Resolves view names to protected .jsp resources within the /WEB-INF/views directory -->
    <bean class="org.springframework.web.servlet.view.InternalResourceViewResolver">
        <property name="viewClass" value="org.springframework.web.servlet.view.JstlView"/>
        <property name="prefix" value="/"/>
        <property name="suffix" value=".jsp"/>
    </bean>
</beans>

In order to show the OAuth confirmation page, I needed to create src/main/java/org/appfuse/examples/webapp/AccessConfirmationController.java and map it to /oauth/confirm_access. I copied this from one of the sample projects and modified to use Spring's annotations.

package org.appfuse.examples.webapp;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.security.oauth2.provider.ClientAuthenticationToken;
import org.springframework.security.oauth2.provider.ClientDetails;
import org.springframework.security.oauth2.provider.ClientDetailsService;
import org.springframework.security.oauth2.provider.verification.ClientAuthenticationCache;
import org.springframework.security.oauth2.provider.verification.DefaultClientAuthenticationCache;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.servlet.ModelAndView;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.util.TreeMap;

/**
 * Controller for retrieving the model for and displaying the confirmation page
 * for access to a protected resource.
 *
 * @author Ryan Heaton
 */
@Controller
@RequestMapping("/confirm_access")
public class AccessConfirmationController {

    private ClientAuthenticationCache authenticationCache = new DefaultClientAuthenticationCache();
    @Autowired
    private ClientDetailsService clientDetailsService;

    @RequestMapping(method = RequestMethod.GET)
    protected ModelAndView confirm(HttpServletRequest request, HttpServletResponse response) throws Exception {
        ClientAuthenticationToken clientAuth = authenticationCache.getAuthentication(request, response);
        if (clientAuth == null) {
            throw new IllegalStateException("No client authentication request to authorize.");
        }

        TreeMap<String, Object> model = new TreeMap<String, Object>();
        ClientDetails client = clientDetailsService.loadClientByClientId(clientAuth.getClientId());
        model.put("auth_request", clientAuth);
        model.put("client", client);

        return new ModelAndView("access_confirmation", model);
    }
}

This controller delegates to src/main/webapp/access_confirmation.jsp. I created this file and filled it with code to display Accept and Deny buttons.

<%@ page import="org.springframework.security.core.AuthenticationException" %>
<%@ page import="org.springframework.security.oauth2.common.exceptions.UnapprovedClientAuthenticationException" %>
<%@ page import="org.springframework.security.oauth2.provider.verification.BasicUserApprovalFilter" %>
<%@ page import="org.springframework.security.oauth2.provider.verification.VerificationCodeFilter" %>
<%@ page import="org.springframework.security.web.WebAttributes" %>
<%@ taglib prefix="authz" uri="http://www.springframework.org/security/tags" %>
<%@ taglib uri="http://java.sun.com/jsp/jstl/core" prefix="c" %>
<html>
<head>
    <title>Confirm Access</title>
    <link rel="stylesheet" type="text/css" media="all"
          href="http://demo.appfuse.org/appfuse-struts/styles/simplicity/theme.css"/>
    <style type="text/css">
        h1 {
            margin-left: -300px;
            margin-top: 50px
        }
    </style>
</head>

<body>

<h1>Confirm Access</h1>

<div id="content">

    <% if (session.getAttribute(WebAttributes.AUTHENTICATION_EXCEPTION) != null && 
                 !(session.getAttribute(WebAttributes.AUTHENTICATION_EXCEPTION) instanceof UnapprovedClientAuthenticationException)) { %>
    <div class="error">
        <h2>Woops!</h2>

        <p>Access could not be granted.
            (<%= ((AuthenticationException) session.getAttribute(WebAttributes.AUTHENTICATION_EXCEPTION)).getMessage() %>)</p>
    </div>
    <% } %>
    <c:remove scope="session" var="SPRING_SECURITY_LAST_EXCEPTION"/>

    <authz:authorize ifAnyGranted="ROLE_USER,ROLE_ADMIN">
        <h2>Please Confirm</h2>

        <p>You hereby authorize "<c:out value="${client.clientId}" escapeXml="true"/>" to access your protected resources.</p>

        <form id="confirmationForm" name="confirmationForm"
              action="<%=request.getContextPath() + VerificationCodeFilter.DEFAULT_PROCESSING_URL%>" method="POST">
            <input name="<%=BasicUserApprovalFilter.DEFAULT_APPROVAL_REQUEST_PARAMETER%>"
                   value="<%=BasicUserApprovalFilter.DEFAULT_APPROVAL_PARAMETER_VALUE%>" type="hidden"/>
            <label><input name="authorize" value="Authorize" type="submit"></label>
        </form>
        <form id="denialForm" name="denialForm"
              action="<%=request.getContextPath() + VerificationCodeFilter.DEFAULT_PROCESSING_URL%>" method="POST">
            <input name="<%=BasicUserApprovalFilter.DEFAULT_APPROVAL_REQUEST_PARAMETER%>"
                   value="not_<%=BasicUserApprovalFilter.DEFAULT_APPROVAL_PARAMETER_VALUE%>" type="hidden"/>
            <label><input name="deny" value="Deny" type="submit"></label>
        </form>
    </authz:authorize>
</div>
</body>
</html>

Finally, I needed to create src/main/webapp/login.jsp to allow users to login.

<%@ page language="java" pageEncoding="UTF-8" contentType="text/html;charset=utf-8" %>
<%@ taglib uri="http://java.sun.com/jsp/jstl/core" prefix="c" %>
<%@ taglib uri="http://java.sun.com/jsp/jstl/fmt" prefix="fmt" %>
<%@ taglib uri="http://java.sun.com/jsp/jstl/core" prefix="c" %>

<html>
<head>
    <title>Login</title>
    <link rel="stylesheet" type="text/css" media="all"
          href="http://demo.appfuse.org/appfuse-struts/styles/simplicity/theme.css"/>
    <style type="text/css">
        h1 {
            margin-left: -300px;
            margin-top: 50px
        }
    </style>
</head>
<body>
<h1>Login</h1>

<form method="post" id="loginForm" action="<c:url value='/j_security_check'/>">
    <fieldset style="padding-bottom: 0">
        <ul>
            <c:if test="${param.error != null}">
                <li class="error">
                    ${sessionScope.SPRING_SECURITY_LAST_EXCEPTION.message}
                </li>
            </c:if>
            <li>
                <label for="j_username" class="required desc">
                    Username <span class="req">*</span>
                </label>
                <input type="text" class="text medium" name="j_username"
                       id="j_username" tabindex="1"/>
            </li>

            <li>
                <label for="j_password" class="required desc">
                    Password <span class="req">*</span>
                </label>
                <input type="password" class="text medium" name="j_password"
                       id="j_password" tabindex="2"/>
            </li>
            <li>
                <input type="submit" class="button" name="login" value="Login"
                       tabindex="3"/>
            </li>
        </ul>
    </fieldset>
</form>
</body>
</html>

All the changes described in the above section are necessary to implement OAuth if you create a project with AppFuse WS 2.1. It may seem like a lot of code, but I was able to copy/paste and get it all working in an app in under 5 minutes. Hopefully you can do the same. I'm also considering adding it by default to the next version of AppFuse. Now let's look at integrating OAuth into a client to authenticate and retrieve data from this application.

Authenticate and Retrieve Data with Client
I originally thought my GWT OAuth application would provide a nice client. However, after 30 minutes of trying to get GWT 1.7.1 and the GWT Maven plugin (1.1) working with my 64-bit Java 6 JDK on OS X, I gave up. So I opted to use the Ajax Login application I've been using in my recent security tutorials.

In this example, I used OAuth2RestTemplate from Spring Security OAuth. While this works, and works well, I'd still like to get things working with GWT (or jQuery) to demonstrate how to do it from a pure client-side perspective.

To begin, I got the latest source of Ajax Login from GitHub (as of this morning) and made some changes. First of all, I added the Spring Security OAuth dependencies to pom.xml:

<repository>
    <id>spring-milestone</id>
    <url>http://s3.amazonaws.com/maven.springframework.org/milestone</url>
</repository>
...
<dependency>
    <groupId>org.springframework.security.oauth</groupId>
    <artifactId>spring-security-oauth</artifactId>
    <version>1.0.0.M3</version>
    <exclusions>
        <exclusion>
            <groupId>org.springframework</groupId>
            <artifactId>spring-beans</artifactId>
        </exclusion>
        <exclusion>
            <groupId>org.springframework</groupId>
            <artifactId>spring-context</artifactId>
        </exclusion>
        <exclusion>
            <groupId>org.springframework</groupId>
            <artifactId>spring-core</artifactId>
        </exclusion>
    </exclusions>
</dependency>

Then I modified src/main/webapp/WEB-INF/security.xml and added an OAuth Token Service and defined the location of the OAuth server.

<?xml version="1.0" encoding="UTF-8"?>
<beans:beans xmlns="http://www.springframework.org/schema/security"
             xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xmlns:beans="http://www.springframework.org/schema/beans"
             xmlns:oauth="http://www.springframework.org/schema/security/oauth2"
             xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
              http://www.springframework.org/schema/security http://www.springframework.org/schema/security/spring-security-3.0.xsd
              http://www.springframework.org/schema/security/oauth2 http://www.springframework.org/schema/security/spring-security-oauth2.xsd">

...
    <oauth:client token-services-ref="oauth2TokenServices"/>

    <beans:bean id="oauth2TokenServices"
                class="org.springframework.security.oauth2.consumer.token.InMemoryOAuth2ClientTokenServices"/>

    <oauth:resource id="appfuse" type="authorization_code" clientId="ajax-login"
                    accessTokenUri="http://localhost:9000/appfuse-oauth/oauth/authorize"
                    userAuthorizationUri="http://localhost:9000/appfuse-oauth/oauth/user/authorize"/>

Next, I created a Controller that uses OAuth2RestTemplate to make the request and get the data from the AppFuse OAuth application's API. I created src/main/java/org/appfuse/examples/webapp/oauth/UsersApiController.java and filled it with the following code:

package org.appfuse.examples.webapp.oauth;

import org.appfuse.model.User;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.security.core.context.SecurityContextHolder;
import org.springframework.security.oauth2.common.exceptions.InvalidTokenException;
import org.springframework.security.oauth2.consumer.*;
import org.springframework.security.oauth2.consumer.token.OAuth2ClientTokenServices;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.ResponseBody;

import java.util.ArrayList;
import java.util.List;

@RequestMapping("/appfuse/users")
@Controller
public class UsersApiController {

    private OAuth2RestTemplate apiRestTemplate;
    @Autowired
    private OAuth2ClientTokenServices tokenServices;

    private static final String REMOTE_DATA_URL = "http://localhost:9000/appfuse-oauth/api/users";

    @Autowired
    public UsersApiController(OAuth2ProtectedResourceDetails resourceDetails) {
        this.apiRestTemplate = new OAuth2RestTemplate(resourceDetails);
    }

    @RequestMapping(method = RequestMethod.GET)
    @ResponseBody
    public List<User> getUsers() {
        try {
            List users = apiRestTemplate.getForObject(REMOTE_DATA_URL, List.class);
            return new ArrayList<User>(users);
        } catch (InvalidTokenException badToken) {
            //we've got a bad token, probably because it's expired.
            OAuth2ProtectedResourceDetails resource = apiRestTemplate.getResource();
            OAuth2SecurityContext context = OAuth2SecurityContextHolder.getContext();
            if (context != null) {
                // this one is kind of a hack for this application
                // the problem is that the sparklr photos page doesn't remove the 'code=' request parameter.
                ((OAuth2SecurityContextImpl) context).setVerificationCode(null);
            }
            //clear any stored access tokens...
            tokenServices.removeToken(SecurityContextHolder.getContext().getAuthentication(), resource);
            //go get a new access token...
            throw new OAuth2AccessTokenRequiredException(resource);
        }
    }
}

At this point, I thought everything would work and I spent quite some time banging my head against the wall when it didn't. As I was composing an email to the Enunciate users mailing list, I realized the issue. It appeared to be working, but from the server side, and the redirect back to the client was not happening. The Ajax Login app uses UrlRewriteFilter (for pretty URLs) to redirect from /app/* to /$1 and this redirect was losing the code parameter in the URL.

<rule>
    <from>/app/**</from>
    <to last="true" type="redirect">%{context-path}/$1</to>
</rule>

To fix this, I added use-query-string="true" to the root element in src/main/webapp/WEB-INF/urlrewrite.xml:

<urlrewrite default-match-type="wildcard" use-query-string="true">

After making all these changes, I ran mvn jetty:run on both apps and opened http://localhost:8080/appfuse/users in my browser. It all worked and a smile crept across my face. I've checked in the client changes into ajax-login on GitHub and the appfuse-oauth example into AppFuse Demos on Google Code. If you'd like to see this example in action, I'd encourage you to checkout both projects and let me know if you find any issues.

How's the ol' Team Doing?

2010-10-21T09:00:41-06:00

Back in March, I wrote about How We Hired a Team of 10 in 2 Months:

This week, we on-boarded 3 of our final 4 developers. I breathed a big sigh of relief that the hiring was over and we could get back to slinging code and making things happen. As luck would have it, I received an e-mail from my boss on Tuesday that the hiring engine is starting up again and we need to hire 6 more developers. While I'm not anxious to start the Hiring Engine again, I am glad to know it works well and it has helped us build a great team.

We never ended up hiring those additional 6 developers, but we've had quite a ride since March. One of the first commenters on my original post wrote:

I hope that whirlwind of hiring works out for you Matt. But, don't you think it's a bit early to be declaring success?

At that time, we still had a lot of work to do to become a successful team. However, by the end of March, we'd finished our first deliverable - converting a somewhat slow ColdFusion/jQuery/Video webapp into a fast Java/jQuery/Video webapp. The slowness wasn't due to ColdFusion, but mostly performance, caching and YSlow-type optimizations. At that time, we surprised the folks that were in charge of our app. They didn't think we'd finish so fast and it took them awhile to decide what to do with our work.

While we were waiting for the product folks to launch our app in April/May, we decided to experiment with developing some new clients. So we wrote an iPhone app, an Android App, an iPhone/Android/iPad/HTML5 version of our webapp and a Blu-ray client. All of these applications used our webapp's backend RESTful services and we were able to learn the SDKs and implement all the apps in a matter of weeks. In May, we demoed all our clients and got rave reviews from executives. We celebrated that afternoon with a big sigh of relief.

The glowing from our many-clients demo was short-lived. A week later, we were asked to enhance our iPad app to include TV-Remote type features, namely channel-changing and DVR functionality. After freaking out and trying to figure out how to deliver such an app in a week, the demo was rescheduled and we were afforded 2 weeks to build the app. After much frantic development, we were able to complete the app in time and the demo was published to YouTube a couple months later.

In June, if you asked me if we were a successful team, I would've definitely said "Yes!" We'd been asked to develop apps for 3 different demos and we delivered on-time.

The remainder of June and July we slipped into a bit of limbo where we weren't asked to develop anything, but simply maintain and enhance the stuff we'd already developed. After a few weeks of doing this, several of us began to wonder if the apps we'd developed would ever see the light of day. We expressed this concern to our VP and a new idea was hatched: The 60-Day Push.

The 60-Day Push was designed to eliminate politics and meetings and allow us to develop a 3-screen (PC, iPad, TV) experience for our video content. We decided to aim high and try to complete most of the work in 30-days, so we could do executive demos of our progress. We started all our applications from scratch, split into Services, Portal, TV and iPad teams and worked with Method to implement a slick design for all our apps. I'm proud to say we delivered yet again and there were many proud moments as we demoed to the top executives of the company.

In early September, after doing several demos, we were approved to launch and we've been working towards that goal ever since. We feel we have several weeks of work to coax our DemoWare into RealWorldWare, but the momentum is there and the end of the tunnel is in sight.

As a further sign of our success, we're moving into a new office in LoDo next week. This also means the End of an Era, where the Raible Designs' office across from Forest Room 5 will cease to exist. We (Goodwill, Scotty, Country Bry and I) first moved into this office while working for Evite in April 2009. It's been a fantastic location, a facilitator of extensive collaboration and host to many dart games, FACs, bird-dog spottings and way too many Mom Jokes.

As part of our transition, I'll be looking for renters to fill out the rest of my lease (through March 2010). If you're looking for a sweet location for 5-6 people in Denver's Highlands (near the REI store downtown), please let me know.

Also, there's a good chance my team will continue to grow as we move our apps into production and start ramping up for millions of users. If you're a strong Web or Java developer with social skills and a Ditchdigger attitude, I'd love to hear from you. We probably won't be hiring until January, but that doesn't mean we can't start talking now.

Developing Rich Web Service APIs with Java

2010-03-18T17:51:00-06:00

This afternoon, I attended Ryan Heaton's talk on Developing Rich Web Service APIs with Java. I've always admired Ryan's work and what he's done with Enunciate. Below are my notes from his talk.

We've come a long way from the WS (SOAP) <-> EJB days. There are many tools that you can use to develop web services today. A Web Service is an API that's accessible over a network via platform-independent protocol. Historical examples of web services (in chronological order):

CORBA
DCOM
RMI
SOAP
REST

A Web Service is composed of code wrapped with a container and it's bound to a contract. The table below shows the types of web services and their equivalent Java standard.

Standard	Technology
SOAP	JAX-WS
REST	JAX-RS
XML	JAXB
JSON	???

JAXB is the standard technology used to marshal XML <-> Java. It's annotation-driven, using @XmlRootElement, @XmlElement and @XmlAttribute to translate Java properties to XML.

JAX-WS is a Java standard, outputs classes as SOAP endpoints and it's annotation-driven. @WebService and @WebMethod are the main annotations. @WebMethod is only needed if you're trying to expose a non-public method.

JAX-RS is a Java standard used to expose REST services. Not surprisingly, it's annotation-driven. The class itself needs to be annotated with @Path. Methods are annotated with @GET, @PUT, @POST or @DELETE. If you need to pass in a parameter to a method, you can add a path parameter like the following:

@Path("/{id}")
@GET
public Person readPerson(@PathParam("id") String id);

To specify the possible mime-type outputs of a method, you can use the @Produces annotation. For example:

@Produces({"application/xml", "application/json"})

You can also do this for input methods with the @Consumes annotation.

For JSON, there's no Java standard. However, there's a number of libraries available, including Jackson, Jettison, GSON, XStream. Personally, I've used Jackson and highly recommend it.

The major players in exposing Java code as JAX-WS and JAX-RS services are Oracle, Spring and JBoss. Oracle (formerly Sun) has implemented the JAX-WS reference implementation and it's called Metro (a.k.a. JAX-WS RI). For JAX-RS, Oracle has the Jersey project. For Spring, the JAX-WS and JAX-RS implementation is Apache CXF. JBoss has JBoss-WS and RESTEasy.

There's also a number of custom containers for exposing web services. For example, AMF (implementations: BlazeDS, GraniteDS), GWT-RPC (GWT), JSON-PRC (DWR), Atom (Abdera), OpenSocial (Shindig).

The contract is a very important part of a web service API. It's composed of an Interface Definition Language (WSDL, WADL JSON Schema), Developer Docs (text, context, preconditions, restrictions), example code and client libraries.

Enunciate allows you to create your contract for your web services. It's a build-time WS Development tool. It generates developer documentation and client-side libraries (for Java, C/C++, Objective C, ActionScript and Ruby). It leverages existing specs (JAX-*) and fails fast at compile time.

From there, Ryan gave us a demo of Enunciate and how it could easily create an API website based on JAX-RS annotations.

I liked Ryan's talk and I'm definitely a fan of Enunciate. While I didn't learn anything new, I think there's a lot of Java developers that don't know about the various standards and how easy it is to develop web services. Hopefully by taking notes from Ryan's talk, I'll get the word out a bit more and make more folks aware of Enunciate. On a related note, Sonatype has a good post on how they documented the Nexus API with Enunciate.

The Cloud Computing Continuum with Bob McWhirter

2010-03-17T16:20:26-06:00

This afternoon, I sat in a Keynote by Bob McWhirter at TSSJS 2010. Bob is the Chief Architect of Cloud Computing at Red Hat. Bob is "The Despot" at Codehaus, helped start Groovy and founded Drools. Below are my notes from his talk.

The cloud is not an either/or question. A scorched Earth strategy is not necessary to be in the cloud. The cloud is a continuum of technologies, many of those that you're already familiar with. You're used to web, messaging and storage. You can add just a little bot of cloud to your application at a time.

So what is the cloud?

It's the next logical step in application delivery.

How did we get to the cloud? First, we jammed a lot of servers into a closet (back in the .com days). That's a lot of stuff to deal with if all you want to do is deliver an internet-based service. Then we did some colo. The problem with colo was that it was still our stuff. Then we leased managed servers. With the cloud, we lease what appears to be servers (or services). This is a path we've all been taking to abstract the annoying bits away from hosting an internet-based service.

The typical diagram of a cloud has IaaS, PaaS and PaaS. IaaS abstracts away hardware. PaaS abstracts away services. SaaS abstracts away software.

Tenants of the cloud:

The illusion of infinite resources.
On-demand acquisition and provision.
Someone else's responsibility.

Magicians provide the illusion of pulling a card out of your ear, but they don't actually do it. Because there are limits of what we can do. With on-demand requisitioning, there's no more purchase orders. There's no more waiting for delivery. No waiting for IT to rack it. Or install it. By having someone else manage your hardware, it's not your problem. There's clear lines of responsibility. This intentionally limits visibility into the implementation. Hopefully it's best-of-breed.

The Cloud is really just a mindset. Virtualization makes it affordable to make mistakes. Repeatedly. Each service is managed as an autonomous entity. Services are designed for scalability.

Architecture 101 give us a presentation layer, a business logic layer and a persistence layer. You can scale it by putting many instances on many servers. However, the truth is that most apps are a big ball of mud. You need to get your application in order, then figure out which parts really needs to scale. Is the web tier really where we have trouble? Painting HTML isn't all that hard.

You don't have to put everything in the cloud. The cloud is not a revolution. We still need containers for web things, messaging things, storage of things and other things. Containers still exist, they're just in the cloud.

From container to cloud
When you have an application and a database, there's not much to abstract away into the cloud. However, as soon as you notice your DB is slow, you'll add caching. Once you hit your machine's memory limit, you discover distributed caching systems. Once you've done this, you'll discover that the network isn't all that slow, especially compared to spinning disks. Then you realize that you don't even need the storage and you can keep everything in cache. From there, you move to a RESTful caching service. Then you call it a NoSQL Service so you can be buzzword-compliant.

As far as caching services, there's many available, but a lot of them are startups. However, the cloud does not have to mean external 3rd-party providers. Remember DBAs? It was their job to view the database as a service and to protect it. There's a good chance that we'll end up with DBAs for services. By turning caching into an independent service, it becomes subject to economies of scale.

The cloud is a mindset. It's an SOA approach that allows groups to specialize and optimize for scale.

So how do you implementing a local cloud? We already have great caching technologies. Bob, with his JBoss hat on, recommends Infinispan and its REST API. To get around network latency, you can still run your cloud on your LAN. You're essentially trading disk traffic for network traffic. You can use many hands (e.g. MapReduce) to make things happen quickly.

Besides caching, there's a number of other services that can be put in the cloud. For example:

Messaging (REST-MQ
Scheduling
Security & Identity (OASIS, etc)
Computation (query & otherwise)
Transactions (REST-TX)
Business Process Management
Telephony (VOIP, SMS)

Your application becomes a stack of services, instead of a stack of software.

Higher-order business-logic services are also subject to cloudification. Ultimately, going into the cloud is not rocket science. The cloud does not turn everything on its head. We don't have to wait for new companies to develop new technologies. Startups may be the best consumers of the cloud, but might not be the best providers. We'll let the startups assume the risks for now. Until the 3rd party services have been proven, it's probably best to build your own cloud of services.

More than anything, the cloud has caused a return to fundamentals. Once your app is a collection of services, you can easily pick and choose what pieces you want to put in the cloud. This is where we're headed, whether you like it or not. As more apps become global-scape, you're going to run into these issues. By adopting a cloud mindset and architecture, your app can be prepared for scalability issues.

For more communication with Bob and his Cloud Evangelism, checkout StormGrind or follow him on Twitter.

Personally, I really enjoyed Bob's talk. It was by far the best keynote today, mostly because he told a story and did it elegantly with a nice-looking presentation. Not only that, but he provided the audience with a practical view of the cloud and ways that we can start using it in our applications today.

Update: You can find the slides from this session on SlideShare.

Developing and Testing GWT Client Services

2009-10-21T06:55:17-06:00

Earlier this week, Hiram Chirino released RestyGWT, a GWT generator for REST services and JSON encoded data transfer objects. You can read more about it in Hiram's post RestyGWT, a Better GWT RPC??. First of all, I'm impressed with RestyGWT because provides something I've always wanted with GWT: the ability to call RESTful services and get a populated POJO in your callback, much like AsyncCallback provides for RPC services.

RestyGWT also allows you to easily create services using only interfaces and JAX-RS annotations. For example:

import javax.ws.rs.POST;
...
public interface PizzaService extends RestService {
    @POST
    public void order(PizzaOrder request, MethodCallback<OrderConfirmation> callback);
}

After taking a brief look at RestyGWT, I thought it'd be interesting to share how I develop and test GWT client services.

Developing GWT Client Services
Writing services in a GWT application can be helpful when you're using MVP, especially since you can EasyMock them in a test. On my GWT projects, I've often used overlay types because they allow me to write less code and they make parsing JSON super simple. I've had issues testing my presenters when using overlay types. The good news is I think I've figured out a reasonable solution, but it does require using GWTTestCase. If RestyGWT supported overlay types, there's a good chance I'd use it, especially since its integration tests seem to require GWTTestCase too.

Rather than using callbacks in my presenters, I try to only use them in my service implementations. That way, my presenters don't have to worry about overlay types and can be tested in a JUnit-only fashion. The callbacks in my services handle JSON parsing/object population and fire events with the populated objects.

GWT's RequestBuilder is one option for communicating with RESTful services. The Development Guide for HTTP Requests explains how to use this class. To simplify REST requests and allow multiple callbacks, I'm using a RestRequest class, and a number of other utility classes that make up a small GWT REST framework (created by a former colleague). RestRequest wraps RequestBuilder and provides a Fluent API for executing HTTP requests. Another class, Deferred, is a GWT implementation of Twisted's Deferred.

As part of my service implementation, I inject an EventBus (with GIN) into the constructor and then proceed to implement callbacks that fire Events to indicate loading, saving and deleting has succeeded. Here's an example service:

public class ConversationServiceImpl implements ConversationService {
    private EventBus eventBus;

    @Inject
    public ConversationServiceImpl(EventBus eventBus) {
        this.eventBus = eventBus;
    }

    public void getConversation(String name) {
        Deferred<Representation> d =
                RestRequest.get(URLs.CONVERSATION + "/" + URL.encode(name)).build();

        d.addCallback(new Callback<Representation>() {
            public void onSuccess(Representation result) {
                Conversation conversation = convertResultToConversation(result);
                eventBus.fireEvent(new ResourceLoadedEvent<Conversation>(conversation));
            }
        });

        d.run();
    }

    public void saveConversation(Conversation conversation) {
        Deferred<Representation> d = RestRequest.post(URLs.CONVERSATION)
                .setRequestData(conversation.toJson()).build();
        
        d.addCallback(new Callback<Representation>() {
            public void onSuccess(Representation result) {
                Conversation conversation = convertResultToConversation(result);
                eventBus.fireEvent(new ResourceSavedEvent<Conversation>(conversation));
            }
        });

        d.run();
    }

    public void deleteConversation(Long id) {
        Deferred<Representation> d =
                RestRequest.post(URLs.CONVERSATION + "/" + id).build();

        d.addCallback(new Callback<Representation>() {
            public void onSuccess(Representation result) {
                eventBus.fireEvent(new ResourceDeletedEvent());
            }
        });

        d.run();
    }

    /**
     * Convenience method to populate object in one location
     *
     * @param result the result of a resource request.
     * @return the populated object.
     */
    private Conversation convertResultToConversation(Representation result) {
        JSOModel model = JSOModel.fromJson(result.getData());
        return new Conversation(model);
    }
}

In the saveConversation() method you'll notice the conversation.toJson() method call. This method uses a JSON class that loops through an objects properties and constructs a JSON String.

public JSON toJson() {
    return new JSON(getMap());
}

Testing Services
In my experience, the hardest part about using overlay types is writing your objects so they get populated correctly. I've found that writing tests which read JSON from a file can be a great productivity boost. However, because of overlay types, you have to write a test that extends GWTTestCase. When using GWTTestCase, you can't simply read from the filesystem. The good news is there is a workaround where you can subclass GWTShellServlet and overwrite GWT's web.xml to have your own servlet that can read from the filesystem. A detailed explanation of how to do this was written by Alex Moffat in Implementing a -noserver flag for GWTTestCase.

Once this class is in place, I've found you can easily write services using TDD and the server doesn't even have to exist. When constructing services, I've found the following workflow to be the most productive:

Create a file with the expected JSON in src/test/resources/resource.json where resource matches the last part of the URL for your service.
Create a *ServiceGwtTest.java and write tests.
Run tests to make sure they fail.
Implement the service and run tests to ensure JSON is getting consumed/produced properly to/from model objects.

Below is the code for my JsonReaderServlet.java:

public class JsonReaderServlet extends GWTShellServlet {

    public void service(ServletRequest servletRequest, ServletResponse servletResponse)
            throws ServletException, IOException {

        HttpServletRequest req = (HttpServletRequest) servletRequest;
        HttpServletResponse resp = (HttpServletResponse) servletResponse;

        String uri = req.getRequestURI();
        if (req.getQueryString() != null) {
            uri += "?" + req.getQueryString();
        }

        if (uri.contains("/services")) {
            String method = req.getMethod();
            String output;

            if (method.equalsIgnoreCase("get")) {
                // use the part after the last slash as the filename
                String filename = uri.substring(uri.lastIndexOf("/") + 1, uri.length()) + ".json";
                System.out.println("loading: " + filename);
                String json = readFileAsString("/" + filename);
                System.out.println("loaded json: " + json);
                output = json;
            } else {
                // for posts, return the same body content
                output = getBody(req);
            }

            PrintWriter out = resp.getWriter();
            out.write(output);
            out.close();

            resp.setStatus(HttpServletResponse.SC_OK);
        } else {
            super.service(servletRequest, servletResponse);
        }
    }

    private String readFileAsString(String filePath) throws IOException {
        filePath = getClass().getResource(filePath).getFile();
        BufferedReader reader = new BufferedReader(new FileReader(filePath));
        return getStringFromReader(reader);
    }

    private String getBody(ServletRequest request) throws IOException {
        BufferedReader reader = new BufferedReader(new InputStreamReader(request.getInputStream()));
        return getStringFromReader(reader);
    }

    private String getStringFromReader(Reader reader) throws IOException {
        StringBuilder sb = new StringBuilder();
        char[] buf = new char[1024];
        int numRead;
        while ((numRead = reader.read(buf)) != -1) {
            sb.append(buf, 0, numRead);
        }
        reader.close();
        return sb.toString();
    }
}

This servlet is mapped to <url-pattern>/*</url-pattern> in a web.xml file in src/test/resources/com/google/gwt/dev/etc/tomcat/webapps/ROOT/WEB-INF.

My Service Test starts by getting an EventBus from GIN and registering itself to handle the fired events.

public class ConversationServiceGwtTest extends AbstractGwtTestCase
        implements ResourceLoadedEvent.Handler, ResourceSavedEvent.Handler, ResourceDeletedEvent.Handler {
    ConversationService service;
    ResourceLoadedEvent<Conversation> loadedEvent;
    ResourceSavedEvent<Conversation> savedEvent;
    ResourceDeletedEvent deletedEvent;

    @Override
    public void gwtSetUp() throws Exception {
        super.gwtSetUp();
        DesigntimeGinjector injector = GWT.create(MyGinjector.class);
        EventBus eventBus = injector.getEventBus();
        service = new ConversationServiceImpl(eventBus);
        eventBus.addHandler(ResourceLoadedEvent.ENGINE, this);
        eventBus.addHandler(ResourceSavedEvent.ENGINE, this);
        eventBus.addHandler(ResourceDeletedEvent.ENGINE, this);
    }

    @SuppressWarnings("unchecked")
    public void onLoad(ResourceLoadedEvent event) {
        this.loadedEvent = event;
    }

    @SuppressWarnings("unchecked")
    public void onSave(ResourceSavedEvent event) {
        this.savedEvent = event;
    }

    public void onDelete(ResourceDeletedEvent event) {
        this.deletedEvent = event;
    }
}

After this groundwork has been done, a test can be written that loads up the JSON file and verifies the objects are populated correctly.

public void testGetConversation() {

    service.getConversation("test-conversation");

    Timer t = new Timer() {
        public void run() {
            assertNotNull("ResourceLoadedEvent not received", loadedEvent);
            Conversation conversation = loadedEvent.getResource();
            assertEquals("Conversation name is incorrect","Test Conversation", conversation.getName());

            assertNotNull("Conversation has no channel", conversation.getChannel());
            assertEquals("Conversation has incorrect task size", 3, conversation.getTasks().size());

            convertToAndFromJson(conversation);
            finishTest();
        }
    };

    delayTestFinish(3000);
    t.schedule(100);
}

private void convertToAndFromJson(Conversation fromJsonModel) {
    Representation json = fromJsonModel.toJson();
    assertNotNull("Cannot convert empty JSON", json.getData());

    // change back into model
    JSOModel data = JSOModel.fromJson(json.getData());
    Conversation toJsonModel = new Conversation(data);
    verifyModelBuiltCorrectly(toJsonModel);
}

private void verifyModelBuiltCorrectly(Conversation model) {
    assertEquals("Conversation name is incorrect", "Test Conversation", model.getString("name"));
    assertEquals("Conversation has incorrect task size", 3, model.getTasks().size());
    assertEquals("Conversation channel is incorrect", "Web", model.getChannel().getString("type"));
}

For more information on the usage of the Timer, finishTest() and delayTestFinish(), see GWTTestCase's javadoc.

The tests for saving and deleting a resource look as follows:

public void testSaveConversation() {
    Conversation conversation = new Conversation().setName("Test").setId("1");

    List<Task> tasks = new ArrayList<Task>();
    for (int i = 1; i < 4; i++) {
        tasks.add(new Task().setName("Task " + i));
    }
    conversation.setTasks(tasks);

    System.out.println("conversation.toJson(): " + conversation.toJson());

    assertTrue(conversation.toJson().toString().contains("Task 1"));

    service.saveConversation(conversation);

    Timer t = new Timer() {
        public void run() {
            assertNotNull("ResourceSavedEvent not received", savedEvent);
            finishTest();
        }
    };

    delayTestFinish(3000);
    t.schedule(100);
}
  
public void testDeleteConversation() {
    service.deleteConversation(1L);

    Timer t = new Timer() {
        public void run() {
            assertNotNull("ResourceDeletedEvent not received", deletedEvent);
            finishTest();
        }
    };

    delayTestFinish(3000);
    t.schedule(100);
}

Summary
This article has shown you how I develop and test GWT Client Services. If RestyGWT supported overlay types, there's a good chance I could change my service implementation to use it and I wouldn't have to change my test. Robert Cooper, author of GWT in Practice, claims he has a framework that does this. Here's to hoping this article stimulates the GWT ecosystem and we get a GWT REST framework that's as easy to use as GWT RPC.

Update: Today I enhanced this code to use Generics-based classes (inspired by Don't repeat the DAO!) for the boiler-plate CRUD code in a service. In a nutshell, a service interface can now be written as:

public interface FooService extends GenericService<Foo, String> {
 
}

The implementation class is responsible for the URL and converting the JSON result to an object:

public class FooServiceImpl extends GenericServiceImpl<Foo, String> implements FooService {

    @Inject
    public FooServiceImpl(EventBus eventBus) {
        super(eventBus, "/services/foo");
    }

    @Override
    protected Foo convertResultToModel(Representation result) {
        return new Foo(JSOModel.fromJson(result.getData()));
    }
}

I'm sure this can be further enhanced to get rid of the need to create classes altogether, possibly leveraging GIN or some sort of factory. The parent classes referenced in this code can be viewed at the following URLs:

There's also a GenericServiceGwtTest.java that proves it all works as expected.

My Experience with Java REST Frameworks (specifically Jersey and CXF)

2009-08-27T14:20:51-06:00

Recently I was tasked with developing a server-side REST strategy for a client. When I started working with them, they were using GWT RPC for exposing services. They wanted to move to RESTful services to allow for a more open platform, that multiple types of clients could talk to. There were interested in supporting SOAP and GWT RPC as well, but it wasn't required. They are using Spring, a custom namespace (for easily creating remote services) and an HTML documentation generator to expose their API in human readable form.

When I first starting developing, I chose to try Enunciate. From Enunciate's homepage:

Enunciate is an engine for creating, maintaining, and deploying your rich Web service API on the Java platform. If that sounds complicated, it's not. All you have to do is define your service interfaces in Java source code. ... Then invoke Enunciate.

Sounds pretty sweet, eh? At first glance, the things I liked about Enunciate were:

The ability to generate multiple endpoints (and clients).
Generates nice-looking documentation.
Allows selecting different frameworks (for example, CXF instead of JAX-WS RI).

Initially, the hardest part of using Enunciate was integrating it into my project. This was somewhat related to having a multi-module project, but moreso related to getting the settings right in my enunciate.xml file. After getting everything working, I encountered a few Spring wiring issues with the GWT Endpoints and with Jersey's Spring support.

The good news is I believe most of these issues were related to my project and how it proxies Spring beans that Jersey couldn't find. Jersey's Spring support only supports annotations at this time, so I was unable to tell it the proxied bean names via XML (or even its own annotations). I'm sure this problem is solvable, but after struggling with it for a day or two, I decided to give up on Enunciate. I had to get something working, and fast.

At this point, I was back to the drawing board. I knew there were plenty of good Java REST frameworks available, but Spring and CXF (for SOAP) were already available dependencies in my project, so I chose that route. I was able to get something up and running fairly quickly with this tutorial and CXF's JAX-RS documentation. I ran into an Invalid JSON Namespace issue, but was able to solve it by adding a custom namespace.

It's not all rosy though, there are still a couple CXF issues I haven't solved:

I'd like to remove the namespace prefixes on returned JSON. I've never seen JSON with "prefix." notation before and it doesn't seem right.
I'm unable to return String[] from services.

While CXF does take a bit of XML for each service, it's kinda slick in that it only requires you to annotate your service interfaces. It also generates documentation for your services (WSDL and WADL), but it's not as pretty as Enunciate's. To solve this, I added Enunciate for documentation only. To make Enunciate work, I did have to add some annotations to my service implementation classes and parse the generated HTML to fix some links to CXF's services, but ultimately it works pretty well.

In the end, I recommended my client not use Enunciate for generating endpoints. This was primarily related to their unique Spring configuration, but also because I was able to easily use the same classes for REST, SOAP and GWT Endpoints. However, I will continue to keep my eye on Enunciate. It could be very useful for the upcoming appfuse-ws archetype. I'm also eager to see better GWT support and the ability to generate Overlay types in future releases.

Enhancing Evite.com with GWT and Grails

2009-06-15T07:41:37-06:00

On my LinkedIn Profile, it says my current gig is a SOFEA consultant at a stealth-mode startup.

SOFEA Consultant, Stealth Mode Startup, Los Angeles, CA. December 2008 -- Present.

OK, I lied. It's not a startup, it's a well-known company that helps you plan parties. For the last 5+ months, my UI team from LinkedIn has been working with Evite.com to enhance portions of their site with a SOFEA architecture.

In January, we started evaluating Ajax Frameworks and came to the conclusion that GWT was right for us. After we chose the UI framework, other team members chose Grails and memcached to develop scalable RESTful services. The architecture we implemented involves using GWT's RequestBuilder to talk to Grails' services, which cache almost all their JSON output in memcached.

To see an example of a feature we developed with GWT, see Evite's Design Gallery. I personally worked on this feature and very much enjoyed becoming a GWT aficionado in the process. GWT's zero-turnaround feature made doing pure client-side work a lot of fun. It's definitely something I'd like to continuing doing at my next gig.

Everyone from Evite is very happy with what we've been able to do with GWT and Grails. We have a stateless architecture and are quickly able to develop both client-side and server-side features. We've learned to scale the client by using out-of-the-box GWT components. We've scaled Grails by caching as much as possible. We serve up Ads and Analytics using the same JavaScript mechanisms that traditional server-side frameworks use.

At the end of this month, my gig with Evite comes to an end. I'll be spending a few weeks at my family's cabin in Montana and then it's on to the next big thing. What's the next big thing? I'm not sure yet, hence the reason for writing this. If you're looking to develop a GWT application, introduce a SOFEA architecture at your company, or simply adopt some open source frameworks, I'd love to help out. Drop me a line and let's start a conversation.

GWT and AppFuse

2009-03-04T22:50:26-07:00

Someone recently sent me the following e-mail asking about GWT integration in AppFuse.

I see from your blog that you're spending some time with GWT at the moment. What's your plan, are you going to integrate GWT as another UI Option for AppFuse?

The reason I'm asking is that I actually checked out all of the AppFuse code from the svn repository yesterday, with the intention of starting off adding some GWT stuff in there. My intention was to start by getting a basic Maven archetype together for GWT as an AppFuse UI.

However, if you're planning on doing this yourself in the near future, then there's no point in me starting doing it, I'd have to learn how to write archetype's for a start (not that it looks too difficult) but you'd obviously do it much quicker.

Being a good open-source developer, I moved the discussion to the developer mailing list and replied there:

It's likely I'll create a version of AppFuse Light with GWT, but I doubt I'll do it in the near future. I hope to release AppFuse 2.1 first (which will include "light" archetypes). I wouldn't get your hopes up in waiting for me to do the work. However, I'd be happy to assist you in doing it. AppFuse Light is now modular and uses the AppFuse backend.

http://raibledesigns.com/rd/entry/appfuse_light_converted_to_maven

Here's how I believe GWT should be integrated:

Create an appfuse-ws archetype that serves up RESTful services (http://issues.appfuse.org/browse/APF-897).
Create an appfuse-gwt archetype that consumes those services. This archetype would contain a proxy servlet that allows #1 to be on a separate host/port.

In addition to #1, I hope to convert the Struts 2 and Spring MVC archetypes to use those frameworks' REST support.

For #2, we could use SmartGWT or GXT. SmartGWT might be better since Sanjiv is a committer on this project.

I know I've been slacking on AppFuse development, but it is ski season and running to work seems to drain my late-night coding ambitions. With that being said, I'm committed to getting AppFuse 2.1 released by JavaOne (hopefully sooner). I figure it's a good week's worth of work and I'll probably have to do it late at night to find the time. That's OK though, I usually really start to enjoy it once I get into it.

Colorado Software Summit 2008 Wrapup

2008-10-28T23:03:23-06:00

Last week, I attended the Colorado Software Summit in Keystone and had a great time. Not only was the weather beautiful and the food delicious, the sessions I attended were awesome. If you ever get a chance to go to this conference, I highly recommend it. It's like being on vacation and learning with a bunch of old friends.

Yan Pujante also attended this conference and documents his experience, photography and presentations in Colorado Software Summit 2008.

Below is a list of my entries for all the sessions I attended.

For next year, I think the conference should shorten its sessions (from 90 to 60 minutes), invite more speakers and cut the price in half (to $999 per person). How do you think the Software Summit could be improved?

RESTful Web Applications with Subbu Allamaraju

2008-10-24T09:52:02-06:00

Subbu works at Yahoo! developing standards, patterns and practices for HTTP web languages. In the past, he was a web service and Java developer. He was also a standards contributor at BEA and an author of books on Java EE. His current passion is HTTP and REST. Subbu confesses that he's not a web developer, has no interest in the internals of programming models used for web frameworks and he's only interested in the visible aspects of the architecture.

"The Web is Mostly Restful"

Being RESTful in an abstract sense means:

Resources are named by URIs
Resources have representations (Atom, HTML, JSON, XML)
Resources contain contextual links to allow navigation of state
There's a Uniform Interface

In the web today, some resources and URIs are personalized, but most are not. Some depend on sessions, but most do not. The consequence of a personalized UI with a non-unique URI is you cannot rely on browser caching.

The web is full of different representations (HTML, XML, JS, PDF, CSS, Flash). The problem with HTML is you can't tell links that you want a particular representation based of a link. The links are hard-coded to be a particular content-type. However, some media types on responses are ignored. This is often a problem with browsers and whether the user has plugins installed.

The Uniform Interface for the web is HTML and primarily links and forms (GET and POST). There's still some misconceptions (e.g. POST is secure). However, it's not about security, it's about idempotency and safety. You need to make sure you only use POST when you're changing data. POSTs are not repeatable. GET URIs are not always refreshable, which is quite unfortunate. Users shouldn't have to fight the back button.

Caching is a fundamental aspect of the web. Even in a personalized site, most of the content can be cached. The web is read-only for the most part. However, many enterprise web applications don't take advantage of caching. This is fine when there's not that many users, but it's bad when you want to scale to thousands of users. There's several frameworks that use cache-busting and prefer backend caching over HTTP caching. These frameworks are not using the web like they should.

Backend caching (e.g. Memcached) uses a non-uniform interface and you need to explicitly program to it. Frontend/HTTP caching has a uniform interface that's pluggable. Backend caching is generally more expensive to develop and deploy. There are cases where data should be cached on the backend, but you shouldn't focus all on backend caching w/o doing some frontend caching.

With Ajax, you get more opportunities to be RESTful. XMLHttpRequest is another HTTP client that can be programmed to. It has full support for the uniform interface, which allows content negotiation, optimistic concurrency and caching. Cross-domain hacks can be done with <script> and <iframe> to tunnel requests over GET. The W3C has been working for the last two years on how to do cross-domain Ajax w/o using hacks. The problem with current cross-domain implementations is they often use GET for everything, which isn't very RESTful. Subbu recommends using a proxy on the same domain if you do need to talk to other domains. This will allow your Ajax code to remain RESTful.

Web Frameworks
Web development is hard because of all the moving pieces that exist. Because of this, many web frameworks have been created to solve the various problems. In 1997, there were servlets. They provided basic plumbing and closely reflected HTTP/1.1. Servlets provided a poor programming model, but it allowed a lot of frameworks to be built on top of it. We don't use servlets to write applications, only to write application frameworks. The second era came about in 2001 when Action-oriented frameworks became popular. In 2004, JSF and friends came to play. JSF is a component-based framework with known limitations (complex, slow, uses POST for almost everything, Ajax is difficult). These limitations have resulted in a number of third-party patches being developed to solve these issues.

JSF was designed to use the request to create a component tree that maintains state. Unfortunately, the state is not something the developer has control over. It's not the state of the application, it's the state of the components. The client's knowledge of the state is mentioned with a cookie and the server keeps the state in the session. The problem with JSF is you don't have a choice of state in your application - you can't write stateless applications like you can with servlets.

JSF uses overloaded URIs for its resources. When you have one URI with multiple representations, there's no way to tell how a representation was chosen. JSF's compromise is to allow client-side state saving. However, they do this by putting hidden field in the form and requiring POST for navigation.

JSF vs. REST
Basically, these two are at opposite extremes. JSF is focused heavily on a UI component model. The people that developed it misinterpreted how the web works and made some fundamental questionable choices. You can patch it, but you can not fix it.

Web 2.0 Frameworks
GWT is a cross-compilation based framework. You write Java to generate JavaScript (b/c everyone hates writing JavaScript). It mashes client and server code into a single source. These layers communicate using GWT-RPC. Typical RPC concerns do not apply since code generation handles coupling and the client is downloaded from the same application. GWT-PRC does POSTs to the server and uses HTTP like a transport layer. To be fair, GWT does allow you to use a RequestBuilder to use the web like it should be used. This class allows more control over HTTP requests, it supports GET and POST and it allows so-called RESTful layers (GWT-REST and GET-Restlet). GWT is focused heavily on ease-of-use, which is good. It's modeled after RPC and breaks the uniform interface and focuses on backend caching. Unlike JSF, GWT is fixable, but the community tends to use RPC instead of RequestBuilder.

SOFEA has a central promise of SOA. Business logic is a reusable service that changes less often. The presentation application calls those services and changes more often. The nice thing about this type of architecture is it allows a separation of concerns and loose coupling. However, it doesn't embrace REST like it should. Appcelerator is an implementation of SOFEA that has a Ruby on Rails-like usability. However, it uses a SOAP/HTTP style with messaging and POSTs to a single URI. Appcelerator is interesting, but it introduces a different style of coupling. It breaks URI opacity and client deals with POX instead of links.

Conclusion
Don't fight the architecture of the web. Innovate and enhance instead of breaking. If nothing else, break judiciously. As developers, we should demand more from our frameworks and make sure they use the web and HTTP like it should be used.

What's Coming in Spring 3.0

2008-10-22T11:51:12-06:00

This morning, I delivered my What's Coming in Spring 3.0 talk for the 2nd time at Colorado Software Summit. Since there is no Spring 3.0 source code to speak of, I was unable to do the "Choose Your Own Adventure" demo at the end.

The good news is I was able to easily upgrade the Spring Kickstart application from Spring 2.0 to Spring 2.5.5 (using annotations). When 3.0 is released, I hope to update this project to use 3.0 as well as show what I needed to change. If I get ambitious, I might even change the UI to use Flex or Ext JS to show a RESTful client. Below is my presentation - hope you enjoy.

View on Slideshare

Presenting Web Frameworks of the Future Tomorrow in Denver

2008-07-30T21:56:17-06:00

Tomorrow (Thursday) night, I'll be doing an encore presentation of my Web Frameworks of the Future at DeRailed. If you're in Denver and would like to hear me ramble while drinking a beer, join us at Forestroom 5 at 6:30.

After the last few days, I'm happy to report I should be in good enough condition to pull this off. If you're curious to learn more about my experience at OSCON and this presentation, please see my writeup on the LinkedIn Blog.

[OSCON 2008] Web Frameworks of the Future: Flex, GWT, Grails and Rails

2008-07-23T16:25:23-06:00

Below is the presentation I'm delivering at OSCON today. Unfortunately, I had to remove slides on GWT and Flex to fit w/in the 45 minute time limit. I hope to expand this presentation in the future, as well as continue to develop the side project I'm working on using these technologies.

view on slideshare | download

GWT and REST

2008-07-21T10:31:58-06:00

I've posted two message to the GWT Google Group in the last couple of days. However, new member messages are moderated and neither has shown up yet. I'm reposting my questions here in hopes of getting some answers.

Is there a way to easily use a REST backend with GWT? I tried GWT-REST. It works, but it seems to be centered towards Rails (I'm using Grails) and it suffers from the SOP issue.

JSONRequest looks promising for cross-domain support, but I can't get it to work either. The provided examples work, but not my simple hello world that returns:

{"response":"Hello World!"}

Also, the example implementation only has GET support, not PUT, DELETE or POST. I can post my REST backend on the public internet if anyone is interested in seeing the issues I'm having.

Thanks in advance for any advice.

Should we change AppFuse to be Struts 2-specific?

2008-05-29T08:29:44-06:00

Dusty recently posted an interesting idea to the AppFuse developers mailing list:

After thinking/coding/reading for a while I think the more interesting task is: Retool AppFuse to be one or more Struts2 plugins based on various higher level app patterns. (AppFuse Facebook, AppFuse Employee DB, AppFuse Blog, AppFuse Basic LDAP, AppFuse Basic Crowd).

This all comes from the fact, that I have been wanting to refactor the AppFuse web layer for Struts. One of the interesting aspects of AppFuse is that it works pretty much the same across all its web frameworks. It does so with some lowest common denominator abstractions that can be ported and look and work the same across frameworks. I have picked my tool(s): Struts 2 and Ruby On Rails when I want to pretend I am young again. I know Spring MVC, JSF, etc. but I have no desire to build significant apps on those platforms. It's not because they suck and Struts2 rules, it is because I know Struts 2 the best, I am most efficient there and it provides everything I need to build great webapps (Let's not devolve to a framework debate). So, I would rather have a more Struts 2-specific web stack, that really leverages conventions born and raised there. The nice thing about the Struts 2 web stack is that it is complemented nicely by AppFuse's data/service layer, since unlike Grails or Rails, Struts 2 has no data or service layer. [Read More »]

Seems like a good idea to me. What do you think?

Someday I'd like to come up with a "compatibility test" that allows others to improve upon the ideas in AppFuse and develop their stacks independently. A suite of Selenium tests that require extensionless URLs might be a good start.

REST and Seam Talks at Denver's JUG

2008-01-09T20:59:45-07:00

After a long hiatus, I decided to attend the Denver JUG meeting this evening. Tonight there's a couple of interesting talks:

Give It a REST by Brian Sletten
Seam by Norman Richards

I'll do my best to live-blog these sessions, so hopefully you can read along and learn everything I do.

Give It a REST by Brian Sletten
This talk isn't an implementation talk, it's more of a motivational talk. Brian is trying to answer the question "Why do we care?". We care because we have a lot of WS-Dissatisfaction. "Conventional" Web Services are often:

too difficult for non-trivial tasks (real complexity)
too complex for trivial tasks (artificial complexity)

RPC-based Web Services are mythically interoperable and time/process coupled in painful ways. SOAP has largely become popular because of marketing dollars behind in. REST is more like the "hippy" way that has grass-roots support with no corporate sponsor.

What makes SOAP difficult? It's remote procedure calls and its tunneled using existing application protocols (HTTP). Furthermore, there are no nouns (mappings to business terms), only handlers. There are no semantics, only handlers. When you tie yourself to a contract/WSDL, you can have anything back that you want - so as long as its simple XML. This isn't entirely true because a lot of things can be shoved into XML (Word documents that are Base64 encoded).

The main problem with SOAP is it solves a problem that most people don't have. It solves an edge case, rather than the main problem.

Many people say "SOAP is secure and REST isn't".

Why do people believe this? It's because of the long list of SOAP-related security acronyms: XML Encryption, XML Signature, XKMS, SAML, XACML, WS-Security, WS-Trust, XrML. Even if you're using these in your system, there's no proof that your implementation is secure. REST is what we all use on the web with online shopping, etc. We don't seem to have a problem with the security we use everyday on the web, do we?

SAO is an architectural style promoting loose coupling among software participants. Sompanies have rigid definitions of what constitutes a SOA. Many believe that SOAP is an essential piece, but it's not. SOAP 1.2 and Doc Lit are improvements, but are they necessary? Interestingly, 85% of Amazon's users chose REST over SOAP when given the choice.

What is REST? The acronym stands for REpresentational State Transfer. It's an architectural style based on certain constraints designed to elicit properties of scalability and extensibility. It's an idealized notion of how the early web should work and helped drive the way it eventually did work. It's more than just URLs!

Resource-Oriented Computing focuses on information spaces, not code or objects. It focuses on logical connections and reduces complexity by separating actions from nouns. In the history of the web, we started with URLs that pointed to documents. Eventually, these documents became dynamic and were generated on-the-fly.

URLs are locations on the web that are horrible names because they change so much. URIs are good names that have no way of being resolved. Fundamentally, REST is a separation of the parts of the system: Nouns, Verbs and Representation. A Resource (in a REST architecture) can be a file, a service or a concept. It can also have different representations. Resources are named with Resource Identifiers. It's simply the means of naming a resource. It's a standard syntax that allows various schemes. Often known as URIs (or IRIs). It's orthogonal to satisfying the reference and it's one of the missing pieces of "normal" web services.

Examples of Representation include 1) a particular dereferencing of a Resource Identifier to a Resource at a particular time, 2) a byte-stream tagged with metadata or 3) it could change based on request or processing/display capabilities of the client (Firefox vs. WAP).

REST's verbs are design decisions to minimize the the complexity of implementing a system. GET retrieves a resource and always returns the exact same result. It doesn't change anything in the backend system. Because of this, it allows for easier layering of your system - particularly when you introduce caching for GETs. POST is used to create (or update) a Resource. It does not require a "known URI" and it supports the append operation. PUT creates (or updates) a Resource, but requires a "known URI" and also supports an overwrite operation. Lastly, there is DELETE, which removes a Resource. This is not supported in modern browsers. Just because browsers don't support them doesn't mean you can't implement them in your applications.

REST's concepts were developed by Roy Fielding in his thesis. He was trying to create a system that had the following architectural properties: performance, scalability, generality, simplicity and modifiability. REST allows us to create true client-server applications. To satisfy scalability requirements, REST is stateless. All parameters travel with the request and no session information is maintained on the server. This improves scalability through load-balancing and allows visibility of intermediary processors.

One of the first things that becomes a bottleneck in enterprise systems is the database. This works fine if you like paying Oracle. By using REST and HTTP concepts, it's easy to take advantage of a cache. This allows replication of an external data set where it's too large to copy locally. REST allows you to create Layered Systems that allow you to have managed dependencies between layers. Having a RESTful architecture allows you to swap out the backend without changing the front-end and vise-versa.

Now Brian is going to do some demos using NetKernel. He recommends using HTML documents to describe services. The beauty of developing a RESTful system is there's nothing preventing you from appending ?wsdl to your URLs to return SOAP.

Invoking functionality using web-friendly techniques is a very nice way to build web applications.

What is controversial in REST? When you are not dereferencing you should not look at the contents of the URI string to gain other information.. However, if you structure your URLs in your application in a hierarchical manner, people will be able to use URIs in this way.

What isn't controversial? No one believes you should rely on sessions or other state at the application level. They also believe using nouns, not verbs is an excellent idea.

Seam by Norman Richards
Seam isn't just a web framework, it's an integration technology for building applications for the web. It's a technology that takes your persistence and web technologies and unifies them so you have a simpler view of your system.

First of all, what is Seam?

It's a unified component and event model - you access all your components the same way
It has a declarative state with a rich context model
It provides deep integration with minimal glue code
It minimizes configuration, prefers annotations to XML
It allows a freedom of architectures and technologies
It also allows testing of components in context

With Seam, you have a number of technology choices to make. The first is which business component model you want to use (EJB 3 vs. POJO). You might choose EJB 3 if you want more clustering capabilities. For persistence, you can use Hibernate or any JPA implementation. Norman recommends using JPA if possible. You also have a choice of languages: Java or Groovy (and possibly Scala in the future). Seam gives you a choice of web frameworks. All the examples today will use JSF, but there's also support for GWT and 2.1 will have support for Wicket. Also, there's some non-committers developing support for Flex. Norman believes component-based web frameworks are the best way to develop web applications. While I hear this from a lot of folks component-based framework authors, it seems ironic that the "best" way to develop webapps is not the most popular way (PHP, Struts, Rails).

I think it's ironic that there's a REST (no state) and a Seam (it's all about state) talk in the same night. I'm tempted to ask why a stateful architecture is better than a REST one, but I'm not really that interested in the answer. I think your architecture should be determined by the needs of your application, rather than choosing the architecture and then implementing an application with it.

In addition to choices, Seam gives you a number of tools: CRUD generation, Eclipse and NetBeans support and full IDE support in JBoss Tools / JBoss Developer Studio.

Now Norman is doing a demo - starting by creating a new Seam Web Project in Eclipse. He mentions that he's skipped a number of steps: downloading and installing Seam, downloading and installing JBoss and configuring your Seam installation and database in Eclipse. He has two projects in Eclipse - apparently the New Project wizard creates two - one for the application and one for testing. It's strange that the tests don't go in the actual project. The directory structure has src/action and src/model for your Seam components. After doing some simple CRUD, Norman starts JBoss and looks at the master/detail screens it generates.

To make a Java class a Seam component, you usually only have to add a @Name annotation to the class. While trying to show us how the Authenticator/login works, Norman tried to open components.xml. Unfortunately, this crashed Eclipse and 30 seconds later - following a bevy of "file does not exist" errors, we're back in action. To inject dependencies in a Seam environment, you use the @In annotation. In the example class, here's what the code looks like:

try {
  currentUser = (Person) entityManager.createQuery("select p from Person p where" + 
       " p.email = #{identity.username} and p.password=#{identity.password}").getSingleResult();
} catch (NoSuchEntityException e) {
  currentUser = null;
}

This example appears to encourage title coupling with the data layer, rather than loose coupling (i.e. a DAO layer). I'm sure Seam doesn't prevent you from developing a more loosely coupled application.

Bug Alert: The <h:messages> tag looks like it always has an <li> - even when there's only one message. Struts 2 has the same issue with its <s:errors> tag. It's disappointing that so many Java framework developers don't have more attention to HTML details.

Probably the coolest part of Norman's demo is how Eclipse auto-synchronizes with JBoss so he never has to start/stop anything when he changes Facelets pages or Java classes. Of course, hot deploy should be possible with any web application if you're using Eclipse Web Tools effectively.

I do believe all-in-one starter frameworks like Seam, Rails, Grails and AppFuse are excellent. However, I also believe they're solving a problem that only 10% of companies have. Most companies don't have the ability to start applications from scratch - unless they're a startup. Most companies have an existing infrastructure in place for the backend and they simply need a better web framework to slap a pretty face on it. I don't know the best solution for this, but it seems like a logical choice to RESTify the backend (possibly with a web framework) and then use a modern web framework for the front-end. IMHO, the best web frameworks for a RESTified backend are Flex, GWT and Appcelerator. If nothing else, these appear to be the most hyped for 2008.

Many of the enhancements that Seam has added to the Java web programming model are being pushed back into the Web Beans JSR. With Seam, injection is bi-directional (input and output), is continuous during application invocation and dis-injected after action is done. The whole purpose of this is to let Seam handle the state of your application. By not worrying about storing/maintaining state, you as a developer can concentrate on business logic more and get your applications done faster.

At this point, Norman started talking about Seam's events and how you can use them. Unfortunately, my battery ran out and I drifted off to do some other stuff. I'm sure Seam's Event Model is pretty cool, I just missed it.

Go Light with Apache Struts 2 and REST by Don Brown

2007-11-15T18:12:58-07:00

After attending Dan's talk on REST, I stayed in the same room and listened to Don Brown talk about Struts 2's support for building RESTful applications. Below are my notes from the event.

What is wrong with today's web applications? You're using a modern web framework and you've cleanly separated your presentation and business logic. The biggest problem in modern web applications is Confusing URLs.

A URL should be a resource indicator - not a method invocation. Often, web applications have little or no caching. People use GET to perform data manipulation and POST may or may not change state (especially with JSF). Another big issue with modern web frameworks is there's too many abstraction layers that hide HTTP headers and it's difficult to manipulate them.

Today's applications are "information silos". There's a lot of information in your applications, but it's all buried in HTML, JavaScript and CSS. There's no way to get this information out of your application unless you explicitly expose it.

The answer to many of these problem is REST. It's the Way of the Web. To solve the information silo problem, you can create a single interface that has multiple representation of the same resource. There's one URI for all types of resources - be it XML, JSON or HTML. How does this work w/o modifying the URL? You modify the URL's extension.

Struts 2 has a couple of plugins that make developing RESTful services easier. The first is the Codebehind plugin and the 2nd is the REST plugin. Don is doing a demo with the REST plugin and shows that there's no Struts configuration files needed (no struts.xml and no struts.properties). The only thing that's necessary is to specify an "actionPackages" init-param on the DispatcherFilter in web.xml. This activates the Codebehind plugin that uses conventions to determine the view template's path.

In Don's demo, he's creating an "OrdersController" that implements ModelDriven. After implementing a setId() method (to set the id from the request parameters), a getModel() method (to return the Order object) and implementing a show() method that returns HttpHeaders, Don starts up his server and shows that http://localhost:8080/order/5 returns an HTML page. Changing the URL to end in /5.json returns JSON, /5.xml returns XML.

public HttpHeaders create() {
    service.save(order);
    return new DefaultHttpHeaders("success").setLocationId(order.getId());
}

The Poster Plugin for Firefox is great when you're working with REST services. Don used this plugin to show us that it's possible to post to JSON and get back JSON results. His demo was impressive, especially the fact that there was no XML configuration required for Struts. I also like how the DefaultHttpHeaders class allows you to manipulate headers in a type-safe manner.

To use the REST plugin, you'll want to use Struts 2.1. If you're using Maven, all you need to depend on is struts-rest-plugin. The struts-codebehind-plugin (as well as struts-core) will be pulled in by transitive dependencies.

One disadvantage of REST vs. WS-* is you can't generate client code from a WSDL. You'll have to write your client by hand. However, one advantage of REST is there's already lots of clients - your browser, curl, etc.

The Struts REST Plugin hasn't been officially released, but hopefully will be in Struts 2.1.1. You can checkout the code from SVN using the URL below. The documentation is located here.

http://svn.apache.org/repos/asf/struts/struts2/trunk/plugins/rest

Great talk Don - and excellent work on the REST plugin for Struts. I can't wait to try it out.

Building scalable, reliable, and secure RESTful services with Dan Diephouse

2007-11-15T18:04:22-07:00

This afternoon, I attended Dan Diephouse's talk on RESTful services. Below are my notes from his presentation.

Roy Fielding coined the term REST (REpresentative State Transfer) in a thesis. Everything is a resource and are addressable via URIs. Resources are self descriptive and manipulated with verbs via a uniform interface. We don't want keys - we want links! Resources are hypertext. Hypertext is just data with links to other resources. Data model refers to other application states via links. This is possible because of the uniform interface!

REST and HTTP
There's 5 main HTTP methods: GET, POST, PUT, DELETE and HEAD. Get is cacheable and safe (there's no side effects). POST is an unsafe operation and can't be repeated. HEAD is used to retrieve a resources metadata, without getting the method body.

To create a new resource, you use POST. The server will return an HTTP 201 (Created) with a Location header. After that, you'll do a PUT to the location and you'll get back an HTTP 200. Another option is to have the client generate a unique id and PUT to it straight away - instead of doing the POST/PUT - where the POST generates the unique URL.

The biggest problem with REST is firewalls. Many firewalls don't allow PUT or DELETE. Google fixes this by adding a header that specifies a method override.

One of the constraints of REST is all communication is stateless. Session state is kept on the client. The client is responsible for transitioning to new states. States are represented by URIs. The advantage is this improves visibility, reliability and scalability. You don't need to replicate session state on your services in a cluster.

ETag header
Resources may return an ETag header when it is accessed. On subsequent retrieval of the resource, client sends this ETag header back. The client can then use a "If-None-Match" header with the ETag value to communicate with the server. The server will send back a 304 (Not Modified) with no body if nothing has changed. LastModified is a similar header that servers send back. The client will then send a "If-Modified-Since" header and get a similar result.

REST allows scalability through Caching - a.k.a. "cache the hell out of it". There's 3 types of cache:

Browser
Proxy
Gateway

How does caching actually work? A resource is eligible for caching if:

The HTTP response headers don't say not to cache it
The response is not authenticated or secure
No ETag or LastModified header is present
The cache representation is fresh

Is your cache fresh? Yes, if the expiry time has not been exceeded and the representation was LastModified a relatively long time ago. If it's stale, the remote server will be asked to grab a new copy of the resource and send it back to you.

HEAD allows you to get metadata about a resource without getting the resource itself. You can use it to test that a resource exists, that a link is valid or to check when a resource was last modified.

There's an "Expect 100 Continue" header you can use to query the server to see if it's capable of receiving a message. The nice thing about this is you get client-server communication before sending the message body. An example was provided where you upload an image from your cell phone. You don't want to start sending the message body if authentication is required. Your phone can check for "100 continue" to determine if it should start uploading the file.

For doing batch operations with REST, you can use HTTP connection pipelining. Unfortunately it's broken by some firewalls. Another option is to POST a whole set of data. GData (an extension to the Atom Publishing Protocol) fixes this by allowing you to post a whole bunch of entries at once. Unfortunately, this approach has received a very cold reception from the community.

The Atom Publishing Protocol is a RESTful protocol for building services. You can use it to create, edit, delete entries in a collection. It's an extensible protocol; examples include paging extensions, GData and OpenSearch.

Why should you use APP for your application? Because it provides ubiquitous elements which have meaning across all contexts. You can leverage existing solutions for security (HTTP Auth, WSEE, Google Login, XML Signatures and Encryption). It also eliminates the need for you to write much of the client/server code. Alternatives to APP include HTTPD, Java (Servlets, Restlets, Spring MVC), Ruby on Rails and many others.

Limitations of REST are HTTP is NOT an RPC or message passing system. It also isn't as secure as other solutions. A lot of times, folks just use SSL and basic authentication - which isn't the most secure system.

Dan posted his presentation on his blog if you'd like to download it.

Raible Designs

Building a REST API with JAXB, Spring Boot and Spring Data

XML to Java with JAXB

Spring Data for JPA and REST

Swagger

Liquibase Integration

Accessing API with RestTemplate

Summary

Developing with AngularJS - Part III: Services

Widget Service

Preferences Service

Using $http and Receiving Data

Summary

Secure JSON Services with Play Scala and SecureSocial

Integrating OAuth with AppFuse and its REST API

How's the ol' Team Doing?

Developing Rich Web Service APIs with Java

The Cloud Computing Continuum with Bob McWhirter

Developing and Testing GWT Client Services

My Experience with Java REST Frameworks (specifically Jersey and CXF)

Enhancing Evite.com with GWT and Grails

GWT and AppFuse

Colorado Software Summit 2008 Wrapup

RESTful Web Applications with Subbu Allamaraju

What's Coming in Spring 3.0

Presenting Web Frameworks of the Future Tomorrow in Denver

[OSCON 2008] Web Frameworks of the Future: Flex, GWT, Grails and Rails

GWT and REST

Should we change AppFuse to be Struts 2-specific?

REST and Seam Talks at Denver's JUG

Go Light with Apache Struts 2 and REST by Don Brown

Building scalable, reliable, and secure RESTful services with Dan Diephouse