Matt RaibleMatt Raible is a Web Architecture Consultant specializing in open source frameworks.

The JHipster Mini-Book The JHipster Mini-Book is a guide to getting started with hip technologies today: AngularJS, Bootstrap, and Spring Boot. All of these frameworks are wrapped up in an easy-to-use project called JHipster.

This book shows you how to build an app with JHipster, and guides you through the plethora of tools, techniques and options you can use. Furthermore, it explains the UI and API building blocks so you understand the underpinnings of your great application.

For book updates, follow @jhipster-book on Twitter.

10+ YEARS


Over 10 years ago, I wrote my first blog post. Since then, I've authored books, had kids, traveled the world, found Trish and blogged about it all.

Developing Services with Apache Camel - Part II: Creating and Testing Routes

Apache Camel This article is the second in a series on Apache Camel and how I used it to replace IBM Message Broker for a client. The first article, Developing Services with Apache Camel - Part I: The Inspiration, describes why I chose Camel for this project.

To make sure these new services correctly replaced existing services, a 3-step approach was used:

  1. Write an integration test pointing to the old service.
  2. Write the implementation and a unit test to prove it works.
  3. Write an integration test pointing to the new service.

I chose to start by replacing the simplest service first. It was a SOAP Service that talked to a database to retrieve a value based on an input parameter. To learn more about Camel and how it works, I started by looking at the CXF Tomcat Example. I learned that Camel is used to provide routing of requests. Using its CXF component, it can easily produce SOAP web service endpoints. An end point is simply an interface, and Camel takes care of producing the implementation.

Legacy Integration Test

I started by writing a LegacyDrugServiceTests integration test for the old drug service. I tried two different ways of testing, using WSDL-generated Java classes, as well as using JAX-WS's SOAP API. Finding the WSDL for the legacy service was difficult because IBM Message Broker doesn't expose it when adding "?wsdl" to the service's URL. Instead, I had to dig through the project files until I found it. Then I used the cxf-codegen-plugin to generate the web service client. Below is what one of the tests looked like that uses the JAX-WS API.

@Test
public void sendGPIRequestUsingSoapApi() throws Exception {
    SOAPElement bodyChildOne = getBody(message).addChildElement("gpiRequest", "m");
    SOAPElement bodyChildTwo = bodyChildOne.addChildElement("args0", "m");
    bodyChildTwo.addChildElement("NDC", "ax22").addTextNode("54561237201");
    SOAPMessage reply = connection.call(message, getUrlWithTimeout(SERVICE_NAME));
    if (reply != null) {
        Iterator itr = reply.getSOAPBody().getChildElements();
        Map resultMap = TestUtils.getResults(itr);
        assertEquals("66100525123130", resultMap.get("GPI"));
    }
}

Implementing the Drug Service

In the last article, I mentioned I wanted no XML in the project. To facilitate this, I used Camel's Java DSL to define routes and Spring's JavaConfig to configure dependencies.

The first route I wrote was one that looked up a GPI (Generic Product Identifier) by NDC (National Drug Code).

@WebService
public interface DrugService {
    @WebMethod(operationName = "gpiRequest")
    GpiResponse findGpiByNdc(GpiRequest request);
}

To expose this as a web service endpoint with CXF, I needed to do two things:

  1. Tell Spring how to configure CXF by importing "classpath:META-INF/cxf/cxf.xml" into a @Configuration class.
  2. Configure CXF's Servlet so endpoints can be served up at a particular URL.

To satisfy item #1, I created a CamelConfig class that extends CamelConfiguration. This class allows Camel to be configured by Spring's JavaConfig. In it, I imported the CXF configuration, allowed tracing to be configured dynamically, and exposed my application.properties to Camel. I also set it up (with @ComponentScan) to look for Camel routes annotated with @Component.

@Configuration
@ImportResource("classpath:META-INF/cxf/cxf.xml")
@ComponentScan("com.raibledesigns.camel")
public class CamelConfig extends CamelConfiguration {
    @Value("${logging.trace.enabled}")
    private Boolean tracingEnabled;

    @Override
    protected void setupCamelContext(CamelContext camelContext) throws Exception {
        PropertiesComponent pc = new PropertiesComponent();
        pc.setLocation("classpath:application.properties");
        camelContext.addComponent("properties", pc);
        // see if trace logging is turned on
        if (tracingEnabled) {
            camelContext.setTracing(true);
        }
        super.setupCamelContext(camelContext);
    }

    @Bean
    public Tracer camelTracer() {
        Tracer tracer = new Tracer();
        tracer.setTraceExceptions(false);
        tracer.setTraceInterceptors(true);
        tracer.setLogName("com.raibledesigns.camel.trace");
        return tracer;
    }
}

CXF has a servlet that's responsible for serving up its services at common path. To map CXF's servlet, I leveraged Spring's WebApplicationInitializer in an AppInitializer class. I decided to serve up everything from a /api/* base URL.

package com.raibledesigns.camel.config;

import org.apache.cxf.transport.servlet.CXFServlet;
import org.springframework.web.WebApplicationInitializer;
import org.springframework.web.context.ContextLoaderListener;
import org.springframework.web.context.support.AnnotationConfigWebApplicationContext;

import javax.servlet.ServletContext;
import javax.servlet.ServletException;
import javax.servlet.ServletRegistration;

public class AppInitializer implements WebApplicationInitializer {

    @Override
    public void onStartup(ServletContext servletContext) throws ServletException {
        servletContext.addListener(new ContextLoaderListener(getContext()));
        ServletRegistration.Dynamic servlet = servletContext.addServlet("CXFServlet", new CXFServlet());
        servlet.setLoadOnStartup(1);
        servlet.setAsyncSupported(true);
        servlet.addMapping("/api/*");
    }

    private AnnotationConfigWebApplicationContext getContext() {
        AnnotationConfigWebApplicationContext context = new AnnotationConfigWebApplicationContext();
        context.setConfigLocation("com.raibledesigns.camel.config");
        return context;
    }
}

To implement this web service with Camel, I created a DrugRoute class that extends Camel's RouteBuilder.

@Component
public class DrugRoute extends RouteBuilder {
    private String uri = "cxf:/drugs?serviceClass=" + DrugService.class.getName();

    @Override
    public void configure() throws Exception {
        from(uri)
            .recipientList(simple("direct:${header.operationName}"));
        from("direct:gpiRequest").routeId("gpiRequest")
            .process(new Processor() {
                public void process(Exchange exchange) throws Exception {
                    // get the ndc from the input
                    String ndc = exchange.getIn().getBody(GpiRequest.class).getNDC();
                    exchange.getOut().setBody(ndc);
                }
            })
            .to("sql:{{sql.selectGpi}}")
            .to("log:output")
            .process(new Processor() {
                public void process(Exchange exchange) throws Exception {
                    // get the gpi from the input
                    List<HashMap> data = (ArrayList<HashMap>) exchange.getIn().getBody();
                    DrugInfo drug = new DrugInfo();
                    if (data.size() > 0) {
                        drug = new DrugInfo(String.valueOf(data.get(0).get("GPI")));
                    }
                    GpiResponse response = new GpiResponse(drug);
                    exchange.getOut().setBody(response);
                }
            });
    }
}

The sql.selectGpi property is read from src/main/resources/application.properties and looks as follows:

sql.selectGpi=select GPI from drugs where ndc = #?dataSource=ds.drugs

The "ds.drugs" reference is to a datasource that's created by Spring. From my AppConfig class:

@Configuration
@PropertySource("classpath:application.properties")
public class AppConfig {

    @Value("${ds.driver.db2}")
    private String jdbcDriverDb2;

    @Value("${ds.password}")
    private String jdbcPassword;

    @Value("${ds.url}")
    private String jdbcUrl;

    @Value("${ds.username}")
    private String jdbcUsername;

    @Bean(name = "ds.drugs")
    public DataSource drugsDataSource() {
        return createDataSource(jdbcDriverDb2, jdbcUsername, jdbcPassword, jdbcUrl);
    }

    private BasicDataSource createDataSource(String driver, String username, String password, String url) {
        BasicDataSource ds = new BasicDataSource();
        ds.setDriverClassName(driver);
        ds.setUsername(username);
        ds.setPassword(password);
        ds.setUrl(url);
        ds.setMaxActive(100);
        ds.setMaxWait(1000);
        ds.setPoolPreparedStatements(true);
        return ds;
    }
}

Unit Testing

The hardest part about unit testing this route was figuring out how to use Camel's testing support. I posted a question to the Camel users mailing list in early June. Based on advice received, I bought Camel in Action, read chapter 6 on testing and went to work. I wanted to eliminate the dependency on a datasource, so I used Camel's AdviceWith feature to modify my route and intercept the SQL call. This allowed me to return pre-defined results and verify everything worked.

@RunWith(CamelSpringJUnit4ClassRunner.class)
@ContextConfiguration(loader = CamelSpringDelegatingTestContextLoader.class, classes = CamelConfig.class)
@DirtiesContext(classMode = DirtiesContext.ClassMode.AFTER_EACH_TEST_METHOD)
@UseAdviceWith
public class DrugRouteTests {

    @Autowired
    CamelContext camelContext;

    @Produce
    ProducerTemplate template;

    @EndpointInject(uri = "mock:result")
    MockEndpoint result;

    static List<Map> results = new ArrayList<Map>() {{
        add(new HashMap<String, String>() {{
            put("GPI", "123456789");
        }});
    }};

    @Before
    public void before() throws Exception {
        camelContext.setTracing(true);

        ModelCamelContext context = (ModelCamelContext) camelContext;
        RouteDefinition route = context.getRouteDefinition("gpiRequest");
        route.adviceWith(context, new RouteBuilder() {
            @Override
            public void configure() throws Exception {
                interceptSendToEndpoint("sql:*").skipSendToOriginalEndpoint().process(new Processor() {
                    @Override
                    public void process(Exchange exchange) throws Exception {
                        exchange.getOut().setBody(results);
                    }
                });
            }
        });
        route.to(result);
        camelContext.start();
    }

    @Test
    public void testMockSQLEndpoint() throws Exception {
        result.expectedMessageCount(1);
        GpiResponse expectedResult = new GpiResponse(new DrugInfo("123456789"));
        result.allMessages().body().contains(expectedResult);

        GpiRequest request = new GpiRequest();
        request.setNDC("123");
        template.sendBody("direct:gpiRequest", request);

        MockEndpoint.assertIsSatisfied(camelContext);
    }
}

I found AdviceWith to be extremely useful as I developed more routes and tests in this project. I used its weaveById feature to intercept calls to stored procedures, replace steps in my routes and remove steps I didn't want to test. For example, in one route, there was a complicated workflow to interact with a customer's data.

  1. Call a stored procedure in a remote database, which then inserts a record into a temp table.
  2. Lookup that data using the value returned from the stored procedure.
  3. Delete the record from the temp table.
  4. Parse the data (as CSV) since the returned value is ~ delimited.
  5. Convert the parsed data into objects, then do database inserts in a local database (if data doesn't exist).

To make matters worse, remote database access was restricted by IP address. This meant that, while developing, I couldn't even manually test from my local machine. To solve this, I used the following:

  • interceptSendToEndpoint("bean:*") to intercept the call to my stored procedure bean.
  • weaveById("myJdbcProcessor").before() to replace the temp table lookup with a CSV file.
  • Mockito to mock a JdbcTemplate that does the inserts.

To figure out how to configure and execute stored procedures in a route, I used the camel-store-procedure project on GitHub. Mockito's ArgumentCaptor also became very useful when developing a route that called a 3rd-party web service within a route. James Carr has more information on how you might use this to verify values on an argument.

To see if my tests were hitting all aspects of the code, I integrated the cobertura-maven-plugin for code coverage reports (generated by running mvn site).

<build>
    <plugins>
        <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>cobertura-maven-plugin</artifactId>
            <configuration>
                <instrumentation>
                    <excludes>
                        <exclude>**/model/*.class</exclude>
                        <exclude>**/AppInitializer.class</exclude>
                        <exclude>**/StoredProcedureBean.class</exclude>
                        <exclude>**/SoapActionInterceptor.class</exclude>
                    </excludes>
                </instrumentation>
                <check/>
            </configuration>
            <version>2.6</version>
        </plugin>
...
<reporting>
    <plugins>
        <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>cobertura-maven-plugin</artifactId>
            <version>2.6</version>
        </plugin>

Integration Testing

Writing an integration test was fairly straightforward. I created a DrugRouteITest class, a client using CXF's JaxWsProxyFactoryBean and called the method on the service.

public class DrugRouteITest {

    private static final String URL = "http://localhost:8080/api/drugs";

    protected static DrugService createCXFClient() {
        JaxWsProxyFactoryBean factory = new JaxWsProxyFactoryBean();
        factory.setBindingId("http://schemas.xmlsoap.org/wsdl/soap12/");
        factory.setServiceClass(DrugService.class);
        factory.setAddress(getTestUrl(URL));
        return (DrugService) factory.create();
    }

    @Test
    public void findGpiByNdc() throws Exception {
        // create input parameter
        GpiRequest input = new GpiRequest();
        input.setNDC("54561237201");

        // create the webservice client and send the request
        DrugService client = createCXFClient();
        GpiResponse response = client.findGpiByNdc(input);

        assertEquals("66100525123130", response.getDrugInfo().getGPI());
    }
}

This integration test is only run after Tomcat has started and deployed the app. Unit tests are run by Maven's surefire-plugin, while integration tests are run by the failsafe-plugin. An available Tomcat port is determined by the build-helper-maven-plugin. This port is set as a system property and read by the getTestUrl() method call above.

public static String getTestUrl(String url) {
    if (System.getProperty("tomcat.http.port") != null) {
        url = url.replace("8080", System.getProperty("tomcat.http.port"));
    }
    return url;
}

Below are the relevant bits from pom.xml that determines when to start/stop Tomcat, as well as which tests to run.

<plugin>
    <groupId>org.apache.tomcat.maven</groupId>
    <artifactId>tomcat7-maven-plugin</artifactId>
    <version>2.2</version>
    <configuration>
        <path>/</path>
    </configuration>
    <executions>
        <execution>
            <id>start-tomcat</id>
            <phase>pre-integration-test</phase>
            <goals>
                <goal>run</goal>
            </goals>
            <configuration>
                <fork>true</fork>
                <port>${tomcat.http.port}</port>
            </configuration>
        </execution>
        <execution>
            <id>stop-tomcat</id>
            <phase>post-integration-test</phase>
            <goals>
                <goal>shutdown</goal>
            </goals>
        </execution>
    </executions>
</plugin>
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.17</version>
    <configuration>
        <excludes>
            <exclude>**/*IT*.java</exclude>
            <exclude>**/Legacy**.java</exclude>
        </excludes>
        <includes>
            <include>**/*Tests.java</include>
            <include>**/*Test.java</include>
        </includes>
    </configuration>
</plugin>
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-failsafe-plugin</artifactId>
    <version>2.17</version>
    <configuration>
        <includes>
            <include>**/*IT*.java</include>
        </includes>
        <systemProperties>
            <tomcat.http.port>${tomcat.http.port}</tomcat.http.port>
        </systemProperties>
    </configuration>
    <executions>
        <execution>
            <goals>
                <goal>integration-test</goal>
                <goal>verify</goal>
            </goals>
        </execution>
    </executions>
</plugin>

The most useful part of integration testing came when I copied one of my legacy tests into it and started verifying backwards compatibility. Since we wanted to replace existing services, and require no client changes, I had to make the XML request and response match. Charles was very useful for this exercise, letting me inspect the request/response and tweak things to match. The following JAX-WS annotations allowed me to change the XML element names and achieve backward compatibility.

  • @BindingType(SOAPBinding.SOAP12HTTP_BINDING)
  • @WebResult(name = "return", targetNamespace = "...")
  • @ResponseWrapper(localName = "gpiResponse")
  • @WebParam(name = "args0", targetNamespace = "...")
  • @XmlElement(name = "...")

Continuous Integration and Deployment

My next item of business was configuring a job in Jenkins to continually test and deploy. Getting all the tests to pass was easy, and deploying to Tomcat was simple enough thanks to the Deploy Plugin and this article. However, after a few deploys, Tomcat would throw OutOfMemory exceptions. Therefore, I ended up creating a second "deploy" job that stops Tomcat, copies the successfully-built WAR to $CATALINA_HOME/webapps, removes $CATALINA_HOME/webapps/ROOT and restarts Tomcat. I used Jenkins "Execute shell" feature to configure these three steps. I was pleased to find my /etc/init.d/tomcat script still worked for starting Tomcat at boot time and providing convenient start/stop commands.

Summary

This article shows you how I implemented and tested a simple Apache Camel route. The route described only does a simple database lookup, but you can see how Camel's testing support allows you to mock results and concentrate on developing your route logic. I found its testing framework very useful and not well documented, so hopefully this article helps to fix that. In the next article, I'll talk about upgrading to Spring 4, integrating Spring Boot and our team's microservice deployment discussions.

Posted in Java at Sep 30 2014, 10:05:38 AM MDT 9 Comments
Comments:

Hi Matt!

Nice blog post. We are happy Camel works for you and it's fun.

You may could simplify your route a bit by replacing:

.process(new Processor() {
    public void process(Exchange exchange) throws Exception {
        // get the ndc from the input
        String ndc = exchange.getIn().getBody(GpiRequest.class).getNDC();
        exchange.getOut().setBody(ndc);
    }
})

simply with:

.setBody(simple("${body.NDC}"))

Happy riding,
Christian

Posted by Christian Müller on October 01, 2014 at 02:53 PM MDT #

Thanks for the tip Christian! I found that simply using "${body.NDC}" didn't work and I had to use "${body.getNDC}" or "${body.getNDC()}".

Posted by Matt Raible on October 01, 2014 at 03:51 PM MDT #

The ${body.NDC} is a shorthand for invoking a getter, but then Camel requires the naming to be Ndc, eg only 1 starting upper case letter.

So in this case you need to define the full method name, eg getNDC.

Posted by Claus Ibsen on October 06, 2014 at 07:30 AM MDT #

[Trackback] This article is the third in a series on Apache Camel and how I used it to replace IBM Message Broker for a client. I used Apache Camel for several months this summer to create a number of SOAP services. These services performed various third...

Posted by Raible Designs on October 08, 2014 at 08:38 AM MDT #

[Trackback] Welcome to the final article in a series on my experience developing services with Apache Camel . I learned how to implement CXF endpoints using its Java DSL, made sure everything worked with its testing framework and integrated Spring Boot ...

Posted by Raible Designs on October 16, 2014 at 08:48 AM MDT #

Matt, thanks for such nice post.

Could you please share your thoughts regarding the next item:

1) Camel compared to other integration frameworks (Spring Integration, Mule)? Can you say that Camel is the "best" :-)?
2) Also, when should we use integration framework and when ESB? There are a lot of talks, just interesting your thoughts

Posted by Orest on November 15, 2014 at 08:45 AM MST #

I haven't written any applications with Spring Integration or Mule, so I can't can't really compare. I don't believe Spring Integration had a Java DSL until recently. Using no XML for configuration was important to me on this project.

I think it's best to start with an integration framework and see if you can get by with that. I think it's like Tomcat (or an embedded app server) vs. a full-blown Java EE server. Starting small and scaling up to an ESB is likely easier than starting with an ESB and scaling down to an embedded integration solution.

Posted by Matt Raible on November 15, 2014 at 11:12 AM MST #

Hi Matt, just reading thought the article quickly, and you refer to noxml, but you seem to be using two xml files : cxf.xml and pom.xml. I assume the pom.xml could be replaced by using gradle, but how about cxf.xml?

Posted by Martin Flower on January 12, 2015 at 10:45 AM MST #

Martin - the cxf.xml file is from Apache CXF, and imported from one of their JARs. Since I don't have the file in my project - I'm not as concerned about it. For pom.xml - yes, you could use Gradle. Or you could use one of Maven's alternative languages:

https://github.com/takari/maven-polyglot/tree/master/poms

Posted by Matt Raible on January 12, 2015 at 10:49 AM MST #

Post a Comment:
  • HTML Syntax: Allowed