Code Coverage – what are we talking about?

Today an interesting discussion about the definition of code coverage took place in the JMockit Users group. I hope my english was good enough to follow the details and maybe contribute something useful. 😉

There was a bit of a confusion about the kind of coverage analysis JMockIt does for its coverage reports. For me, as an ISTQB certified tester, the german terms Anweisungsüberdeckung, Zweigüberdeckung and Pfadüberdeckung have a clear meaning. It’s based on the ISTQB definitions. The german Wikipedia article Kontrollflussorientierte Testverfahren has a good explanation of all the different coverage analysis. (Including the Bedingungsüberdeckung, which you can have the most fun with. And with fun I mean sleepless nights of whitebox testing… ;-))

Unfortunatly most of the tools doing some kind of code coverage analysis try to explain their metrics without the use of control flow graphs or any other kind of graphical representation. I think this adds to the confusion.

Using real world terms like branch and path don’t help to clear things up either. That 100% branch coverage means empty branches too isn’t very clear for the beginner who thinks of a branch as a real world thing.

After all this discussion shows once again, that you really have to know the metrics you use or chance is good you’re measuring the wrong things.

Testing the coverage metric of JMockIt

After yesterdays article about Code Coverage terminology I thought more about the code coverage metric JMockIt uses. I created a simple class and a unit test to get more insights:

This is as simple as it gets. The example is taken from the Wikipedia page on Kontrollflussorientierte Testverfahren again. For 100% statement coverage you only need one test case, in which y > x. Then all statements are executed and you have 100% statement coverage.

And here is the resulting JMockit coverage report:

As you can see, JMockit tells us that every single statement is executed, exactly 1 time, with this test case. And that’s exactly what I understand as statement coverage and it is in full compliance to the ISTQB terminology. Now, to make my point really clear, let me change the class as follows:

I added the empty ELSE statement that I omitted first. And I run JMockit again to get a new coverage report:

This is not 100% branch coverage, ISTQB certified or not. 😉 My point on this is, if you want to use a tool to verify your testing requirements, make sure that you know what the tool is measuring. And make sure you read the discussion thread in the JMockit users group.

*Update* I created a Cobertura coverage report for CoverMeSimple now:

Easy code coverage reports with JMockIt

In this blog post I want to describe how I use JMockIt not only for stubs and mocks, but for easy generation of code coverage reports while developing. JMockIt is my favorite tool for unit testing, because of it’s ease of use and the many options you get out of this framework. Only recently I decided to try the code coverage report that comes with JMockIt. I was searching for an easy way to monitor my test coverage while continuing development. I didn’t want another tool or another VM running a fancy code review tool. I just wanted to see what my current test cases are covering.

My class & test

For all of you wanting to get working code I give you a code kata that I created earlier. It’s the FizzBuzz kata, and it is already prepared to run with JMockIt.

How to create reports

To create code coverage reports with JMockIt you don’t have to go to hours and hours of tool setup before you get something done. Just drop these JAR in your classpath and you’re setup to go:

  • jmockit.jar
  • jmockit-coverage.jar
  • jmockit-coverage-html[basic full].jar
  • JDK_HOME/lib/tools.jar

When you’re working inside Eclipse I recommend adding the tools.jar as a user library. See the Eclipse documentation for some help with that.

Update: With Version 0.933+ you don’t need the JDK tools.jar nor the jmockit-coverage.jar in your classpath anymore. That reduces the setup to 2 JAR files. Notice that you only need the jmockit-coverage-htmlbasic.jar> if you intend to view the report only in Eclipse. The FULL report has additional information, that might be useful to you, but I prefer fast tests and simple reports while developing. Thanks for the tips, Rogério.

Now run your JUnit test and you get a directory coverage-report with an index.html file in it. Open it in your favorite browser (or with the Eclipse/Java EE internal web browser) and you see a report.

When you dig into the FizzBuzz.java you see the details:

Do you smell something? What is that red line about? Is there code that is redundant or maybe plain wrong? Is the testcase testing the wrong things or taking false results for true? If you stumble upon something like this, this is the point where you notice that quick coverage reports are worth the setup time.

There is also a way of creating the reports from your ant script. But I’ll save that for a later blog post. If you want to try it yourself, go read the official documentation about JVM parameters.

What I think about it

This is as easy as code coverage gets: Covered lines are green, uncovered lines are red. You even get a little counter before each line telling you how many times this line was hit. Searching for performance bottlenecks? Start looking on high line counters. A whole method appearing in red? Examine the preconditions and write a test that calls that method.

Best thing on JMockIt coverage reports is: They don’t slow down your tests very much. You can just drop the JARs in your classpath and continue with your unit testing. If you want to check on your current coverage status, open the report and take a quick look. I think this is easy enough for every developer to integrate into his process. And you are already doing unit tests, right? So it shouldn’t take you more than a few mouse clicks to get a first coverage report…

Google Wave Robot in Java

Heute ging’s mir wieder besser und ich hab mich an meine Google Wave gesetzt. (Noch jemand eine Einladung?) Google Wave ist ein Tool zur Online-Zusammenarbeit, mir kommt es im Moment wie ein Gruppenchat mit erweiterter Textverarbeitung vor. Man kann aber auch Gadgets hinzufügen, also Karten, Buchcover-Anzeigen, Youtube-Videos und anderes. Damit kann man wohl ziemlich viele der in der Zusammenarbeit anfallenden Arbeiten erledigen.

Ich wollte jetzt einen Robot schreiben, quasi ein Programm, das als Benutzer mit in der Wave sitzt und Fragen beantworten kann. Als erstes fiel mir da ein IMDB-Robot ein, der Filmtitel aus dem Text der Wave heraussucht und weitere Details anzeigt. Allerdings scheint die IMDB keine programmatische Abfrage zu unterstützen (und mir ist auch so, als hätte es da vor einigen Wochen Aufregung über die automatische Abfrage gegeben). Glücklicherweise gibt es noch Seiten wie diese, mit der man Abfragen stellen kann. Mein Plan ist also nicht ganz aus der Welt… 😉

Los ging’s aber erstmal mit einem einfachen Hello World-Robot. Google bietet ein sehr schönes Tutorial zur Wave API. Dort ist auch erklärt, wie man sein Eclipse zum Deployment in die Google AppEngine einrichtet – dort muss der Robot laufen. Die AppEngine ist ein Cloud-Service, in dem eigene Programme laufen können. Der Robot selbst ist quasi ein Servlet, das bestimmte Wave-Event erhält und darauf antwortet.

Nachdem das erste Hallo Welt in meiner Wave aufgetaucht war, habe ich den Robot noch ein wenig verändert. Im Moment antwortet er einfach auf Blips mit dem umgekehrten Text.

Der Robot hört bisher auf 2 Events, nämlich die Anmeldung neuer Benutzer (WAVELET_PARTICIPANTS_CHANGED) und das Erzeugen eines Blips (BLIP_SUBMITTED). Die beiden Events müssen in der capabilities.xml eingetragen sein.

Was mir noch nicht 100%ig klar ist, ist die Versionierung des Robots. Einmal gibt es die Version der AppEngine, dann die Version des Robots. Der Robot muss zwingend neu versioniert werden, wenn sich die Events verändern, auf die er hören soll. Bei der Versionierung der AppEngine geht es wohl eher um Releases, also veränderte Funktionalität. Vielleicht finde ich ja dazu noch ein paar schöne Tutorials oder Erklärungen. Bei meinem Testrobot werde ich wohl nicht weiter versionieren, nur bei Eventänderungen den Zähler hochsetzen. Ich müsste mal ausprobieren, ob ich den Robot auch in Groovy schreiben kann, denn damit wäre das Verarbeiten von JSON- oder XML-Antworten der Webservices sehr viel einfacher zu schreiben. Es gibt eine Anleitung für AppEngine Groovy, die werde ich mich als nächstes zu Gemüte führen.