« Script.aculo.us vs.... | Main | Testing with Perform... »
20060203 Friday February 03, 2006

EMMA vs. Cobertura for Code Coverage I'm looking to add code-coverage reporting to AppFuse. The two open source libraries I know of to do this are EMMA (CPL) and Cobertura (ASL). Which one is the better library to use? Both projects seem to be actively developed - and there are AppFuse HowTos for both, so the decision making process is a bit difficult.

Any insights into why one tool (or project) is better than the other is appreciated. Posted in Java at Feb 03 2006, 05:16:35 PM MST 10 Comments

Comments:

Cenqua has open-source licenses for Clover which is really nice but that might be problematic with a product like Appfuse that people then bundle into commercial products. I've heard great things about EMMA and a colleague of mine has been using it for a while and loves it. This is minor, but one of the things I like about Cobertura is that they use green/red bars rather than just numbers like EMMA does. I find it much easier looking at the sample Cobertura report to visually identify the problem spots than looking at the EMMA sample reports.

Posted by Todd Huss on February 03, 2006 at 06:35 PM MST #

I currently use Emma and I've played with Cobertura. I've viewed code from both. Which one works best for you will depend on your needs.

Cobertura is based on JCoverage and uses ASM. The code is very clean and easy to understand. However, it injects a significant amount of bytecode which will have a significant impact on performance.

Emma does its own bytecode manipulation and the source code is not nearly as clean. It injects a small amount of efficient code. Instrumented bytecode will run almost as fast as regular bytecode. I also think it is faster at instrumentation, but I'm not sure about that. Emma does partial line coverage which I find useful.

They both generate nice reports, but I've heard that Cobertura's are nicer. If you want to dig into the source of the tool, go with Cobertura. Otherwise, I'd recommend Emma.

Posted by Michael Slattery on February 03, 2006 at 08:17 PM MST #

The pro that cobertura has that emma does not have is the <cobertura-check> ANT tag. Everyone knows that code coverage is important, but if it is only at 10% and nothing "bad" happens then what is the point of collecting the metric. With the <cobertura-check> tag you can specify that you must have at least a certain percentage of code coverage or else the build will fail. However, note that cobertura is NOT Apache licensed...only the ANT task are. Because cobertura is a derivative of jcoverage, it must be GPL'd. Also, note that EMMA can do the coverage check on jar files.

Posted by Aaron Korver on February 03, 2006 at 10:02 PM MST #

We are using EMMA, but never tried the other one. Another person at work was evaluating different frameworks an chose emma. hth.

Posted by Karsten Voges on February 04, 2006 at 11:12 AM MST #

Hey Matt,
FYI - the license for Cobertura is mostly GPL, only the ant tasks are ASL.
Based on the licenses I'd lean towards EMMA but that is just my $0.02 :-)

Posted by Bill Dudney on February 06, 2006 at 09:55 AM MST #

The "partially covered" feature of EMMA is great.

Posted by JiaYun on February 09, 2006 at 10:33 AM MST #

The EMMA website looks a lot better - but don't let that trick you. Cobertura's reports are a lot nicer looking. One thing that Cobertura has and EMMA does not - is the ability count the number of times each line is run. The javadoc style of Cobertura also makes it more obvrious where the source code is allowing for easier navigation. One application of this that I'm using is to make a quick, easy to read report for code walks. In this case the server code has already been written, and is being used. Having a coverage report lets you walk through which parts of the code are being run at setup, rmiCallA(), rmiCallB() etc. Take a look at the samples... Cobertura's reads easier: http://cobertura.sourceforge.net/sample/ http://emma.sourceforge.net/coverage_sample_a/index.html

Posted by McGinley on February 21, 2006 at 11:12 PM MST #

In my particular case emma is more accurate than cobertura: http://kwakil.blogspot.com/2006/07/cobertura-vs-emma.html

Posted by Tim Azzopardi on July 30, 2006 at 01:02 PM MDT #

emma gives stats on method coverage, which i find extremely helpful in browsing results, even though cobertura has a more "readable" display using bars as well as numbers.

i like cobertura's line counters, that's very helpful in its own right, and i also like emma's indication that some conditions on a line have not been evaluated (shows where logical short-circuits always fired, preventing testing of some conditions)

while having all together would be great, my preference is emma.

Posted by dave goldstein on September 13, 2006 at 08:47 AM MDT #

For what it's worth: any significant EMMA development seems to have ceased (as of Oct 2007). The primary EMMA author is hoping to have time to reignite EMMA development but I wouldn't hold my breath based on the tone of his updates. (Really too bad as I really liked EMMA.) On the other hand, Cobertura development continues at a brisk pace, so if you have problems and/or enhancement requests, they're likely to be heard by the Cobertura team.

Posted by The Wabbit on October 14, 2007 at 08:32 AM MDT #

Post a Comment:
  • HTML Syntax: Allowed
Click me to subscribe
Matt Raible is a Web Architect who enjoys developing applications with open source technologies. Contact me for rates.
« November 2008
SunMonTueWedThuFriSat
      
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
      
Today

Recent Entries

Tag Cloud

abbie acegi appfuse denver flex grails gwt hibernate jack java jsf linkedin maven myfaces oscon rails skiing spring springmvc struts struts2 tapestry travel webframeworks wicket