Ignoring JUnit tests at runtime

Sometimes I want to ignore particular test failures. In fact I don't want the tests to run at all because I know they will fail. Not that they are broken, but because they depend on part of the infrastructure that isn't available right now.

E.g. in the web-validators project, I wanted to have the ability to ignore the integration tests (which test remote validation services integration) when the remote sites aren't reachable.

I could also have used Assume, but somewhat I am not completely satisfied with that solution. The Assume calls go in the test code and

• can't be used at a class level


• aren't as good from a self-documenting perspective


• don't have access to method information, e.g. other annotations


• will execute my test code anyway, which can cause slow-downs, etc (time outs generally take a bit of time to trigger)

I could also have grouped my network dependent tests in a special suite or using a Category, but then I have to enable or disable it myself (i.e. make suites that match my grouping).

I prefer tests that enable themselves automatically.

So I came up with this solution: I have created a new Runner, and a new version of the Ignore annotation called RuntimeIgnore to which I specify optional extra argument(s): the name of a class to instantiate and execute to check whether an ignore condition is satisfied.

The disadvantage of the current patch is that I still have to have my own Runner. I don't like that too much (because to combine multiple enhancements you might have to combine runners) so I submitted an enhancement to JUnit as I merged that approach into the existing JUnit 4 Runner and @Ignore annotation.
Note: the patch isn't fully complete (cleanups, documentation left aside), it's more like a RFC. Feel free to comment.

How could you use this ? Instead of doing something like:

@Category("LinuxIT")
@Test thisTestShouldOnlyRunOnLinux() {
  // my test code
}

and then enable your group of tests using Suites.

or

@Test thisTestShouldOnlyRunOnLinux() {
  if (notOnLinux()) {
    // my test code
  }
}

You can do:

@Ignore(ifTrue=NotOnLinux.class)
@Test thisTestShouldOnlyRunOnLinux() {
  // my test code
}

The same apply to classes, e.g.

@Ignore(if=NotOnLinux.class)
class MyLinuxTests {
  @Test ...
  @Test ...
}

You can also take advantage of DRY principles on Convention over Configuration doing

@Ignore(ifTrue=TestNameDoesntMatchPlatform.class)
@Test onLinux_aTest() {
  // my test code for Linux
}

@Ignore(ifTrue=TestNameDoesntMatchPlatform.class)
@Test onWindows_aTest() {
  // my test code for Windows
}

( in case you haven't followed the above code will not run in code taken from the JUnit official repository ! )

Update 1: after a comment from a JUnit commiter on my pull request/RFC, I made a slightly different version of the patch that doesn't require any change made to JUnit. By using MethodRules intead of a custom Runner, I am achieving almost the same thing. See the change here.