Monday, May 17, 2010

Some JUnit tricks for easier and better test cases

Whenever I start a new project, the first thing I'll write is a stack trace filterer/clarifier.  The second would be several enhancements to JUnit.  For test driven development to be effective, you need good coverage and lots of tests.  For that to happen, tests need to be dead-simple to write, easy to maintain, and fast.  The following enhancements have really helped me towards these goals.

1. Easy Fixtures - Many bugs/regressions will involve a set of complicated data relationships.  Create an XML/JSON/whatever format to easily import large sets of data.  For example, let's say bug #535 comes in: "Exception is thrown when trying to delete users when they are in a group".  You can quickly go out, and create 535.xml:

Then, your test case could be:

Without an extremely easy way to create test fixtures and import them, 99% of us will write far less tests.

2. Automatic transaction handling and application bootstrapping.  Ideally, the example test method I gave above should be all it takes to run the test.  No explicit setUp, tearDown for bootstrapping your environment, setting up transactions, etc.  It is worth investing your time in some means (maybe a AbstractTest superclass?)  to set these things up automatically for the 95% of test cases that require no special environmental setup.  Don't Repeat Yourself.

3. More powerful Set and List comparison - When I do assertEquals(expected, actual) where "expected" and "actual" are Collections with dozens of elements, the last thing I want to hear that they are not equal, maybe with a toString of both.  So I always implement a assertSetsEqual and assertListsEqual.  assertSetsEqual compare the two collections, where order doesn't matter.   Here's a sample of assertSetsEquals:


The List comparison, where order matters, requires more work to produce a nice message, but it's worth it:

Expected: [a, b, c, d, e, f, g, h, i, j, k]
Actual:   [a, b, c, e, f, g, h, i, j, k, l]
Missing 'd' @ 3
Unexpected l @ 10


4. Type-checking assertEquals - Due to limitations of Java's type system, assertEquals takes two parameters of type Object.  There can be no compile-time checking that the two arguments are actually of types that can actually have equal objects.  For example:

I discovered org.junit.Assert as I wrote this, much better than the junit.framework.  Why is there two?  Just printing the types goes a long way.  I'm surprised how often an assertEquals LOOKS right, but it happily returns false because you are passing incompatible types in.

5. An Eclipse template for creating a new test case.

Just laziness mostly, but I do think it's important to have that fail("No Asserts") in there from the start.  Too often I'll write a test case as I'm exploring a problem, but then I get distracted and abandon it before I really nail down the asserts.  Without asserts, your test case is mostly just slowing things down.

These little changes seem like overkill in the context of writing a silly little test case.  But if your goal is to write hundreds, I found these tricks to be really worth the investment.

Update #1: In the comments, Crias points out that junit.framework is for backwards compatibity with JUnit 3 and earlier and should be avoided if possible.  I think a custom assertEquals which uses reflection determine if the two arguments even can be equal is worthwhile,  but the new org.junit goes a long way just by printing class names.

Update #2: In the comments, David Karr correctly points out the the test runner will print any exception that bubbles out a test method created with my template.  The reason I took this weird approach is the the Eclipse JUnit runner does not output the trace to the console but to the JUnit View, which I find is a goofy place to inspect the exception.  More importantly, the Eclipse runner won't filter my stack traces, which I'm addicted to.

Update #3: Tym The Enchanter points out that Hamcrest helps with my #3 and #4 points about better asserts.  I know very little about Hamcrest, but here's my naive reaction:

  1. is/equalTo for safer equal to comparisons.  These basically force the type of the "actual" to be a subclass (or the same class) as the "expected".  This goes a long way to reducing dumb-dumb comparisons like new Integer(2) and "2".  In my experience, these are exactly the kinds of comparison errors that slow you down.  It's not perfect, however - it's quite possible that the compile time type of your expected is not a subclass of the actual, but they ARE assignable - see the "Garfield" example below.  Of course you can fall back to more traditional assertEquals if it bites you, and I'm unsure how often I'd see this in real life.  
  2. Set/List comparisons - For sure, Hamcrest beats JUnit asserts here.  However, these comparisons basically fail fast - they only tell you the first problem they find - which could lead you to running the test several times before you get it right.  Tell me all of the problems right off the bat! 

Thanks for the comments!

5 comments:

Crias said...

"I discovered org.junit.Assert as I wrote this, much better than the junit.framework. Why is there two?"

If I'm not mistaken, it's a compatibility thing.

I believe JUnit 3 and earlier used "junit.framework.Assert" for assertions.

In JUnit 4 they moved to "org.junit.Assert", but left "junit.framework.*" so that JUnit 3 tests could run in a JUnit 4 container.

Whenever you're writing new tests, always use org.junit.* and avoid junit.framework.*, and try to convert old tests as soon as possible.

Teja said...

Very nice and informative post. Thanks for the great article. Keep posting.

David Karr said...

Actually, a better template might be this:

public void test() throws Exception {
fail("No asserts");
}

There's no reason to have a catch clause just to print the exception and fail. If the method throws any exception, the test runner will print the exception and fail.

Tym the Enchanter said...

If you want some more type safe checking look at the assertThat(T actual, Matcher<T> matcher) methods from the Hamcrest matchers. There are the basic ones bundled with JUnit and more available from the Hamcrest site.

These use typed Matchers and are generally more flexible that the simple junit.Assert type comparisons allowing you to write a test like

assertThat(testValue, is(expectedValue));

and for collections:

assertThat(collection, containsInAnyOrder(element1, element2, element3))

for order free checking or

assertThat(collection, contains(element1, element2, element3))

for specific order checking. Both will fail if the collection contains an unexpected element.

Finally, the failure messages are generally clearer than the basic JUnit ones.

Java Tutorials said...

great article. I would like to add that we can also the Junit categories feature added in JUnit 4 for categorizing the test cases and executing a particular category only.