Unit / Functional Testing and jUnit

Terence Parr

Last updated: February 7, 2006

Introduction

Unlike mathematics where you can prove that a result is correct, there is no general way to guarantee software is correct. You just have to try out the software under as many conditions as possible to see if it fails. Fix the bug(s). Repeat.

So, we test because Humans have very limited brains and we make lots of mistakes, particularly as the number of lines of code and number of programmers goes up. The complexity goes up in a highly nonlinear fashion.

Besides increasing the quality of the user experience, less buggy software means you will be awakened less often at 3AM because the server has crashed. More robust software is less expensive to modify and maintain. Maintenance is the vast majority of the effort when developing software. Unlike school projects that you never look at again after the end of the semester, real projects live for years or decades. Somebody has to deal with your bugs for a long time--possibly you!

Programmers will often avoid writing tests and performing tests because it is a hassle. jUnit (brought to you by the extreme programming folks that emphasize "testing early, testing often") removes the hassle by providing a very simple set of Java classes that helps you build unit tests. It provides the framework for specifying tests and includes a nice GUI for running the tests and collecting results.

Why do we automate testing? While some applications like GUIs often require human testers, you should automate your tests so that you can run them easily. If you can run them easily and automatically, they can then be integrated into the build process. Automated tests answer the following question very quickly, "did my latest change break anything?"

Automated tests, therefore, let you code with confidence; confidence that you will know when/if you break something when adding new functionality or refactoring. Here is a coding truism: A confident programmer is a fast coder!

There is nothing more terrifying than modifying software that has no tests because you don't know what you might be breaking. You can't clean up the code and you can't extend it. My parser generator ANTLR v2 is like this. I literally refuse to alter its grammar analyzer or code generator for fear of breaking the programs of the many ANTLR users around the world.

In contrast, there is nothing quite like the pleasure of seeing all of your unit tests light up green in the jUnit GUI. It really does give you a good feeling. :)

What else can testing help? A bunch of tests often provides better "documentation" of your code's functionality than does your real printed text documentation. At the very least, the tests will be up-to-date versus your documentation and less likely to be misleading.

When do you test? Test often, such as with each build so that you can find problems right away: this usually tells you precisely what you screwed up. A friend at Microsoft told me that they run all tests on a giant server "build farm" at night and if you have submitted something that fails tests or doesn't compile (shudder), they can ask you to come in early to fix it so it doesn't hold up the whole team.

What do you test? As much as you can given the time constraints and the nature of your software. Make sure that you test not only good input, but bad input as well. With bad input make sure your software not only catches it but executes the appropriate cleanup or error code. Another good idea is to try the boundary conditions. For example, a method that looks at an array, try to get it to hit the last element or first element of the array. Also try on either side of the boundaries.

How do I integrate testing into my debugging process? First, build the test that reproduces the bug. Watch it fail. Then work on the software until that unaltered test passes. Do not fix the software and then build a test that passes! You must positively absolutely see the test go from failure to passing.

Making a test for each bug is a great idea because errors tend to repeat themselves. Also the more tests you have, in principle, your software will get more and more robust with time.

What does testing cost? Testing is often more work than the actual application. Further, it often forces you to refactor your application to create testing hooks so that your testing harness can talk to your application. In my experience, adding testing hooks leads to better organized and more general code.

Testing can take some work in the near term, but in the long run you are faster and get better, more maintainable code.

What is a unit test versus a functional test? A unit test is usually a test on a method not on the overall functionality of a tool or application. For example, testing a method that returns large prime numbers is a unit test but testing the overall encryption algorithm that uses that method is a functional test.

Ok, let's dive into testing. These course notes describe how to begin building tests with jUnit, but does not go into great detail about the class hierarchy etc... (Learning how to test software is an art all by itself). You may find the FAQ and the jUnit Home Page useful for learning the mechanics of testing.

The World Before jUnit

Imagine that you have a Calculator class and you would like to perform some unit tests on the methods. You could make a main method that does the testing:

public class Calculator {
    public int add(int x, int y) { return x+y; }
    public int mult(int x, int y) { return x*y; }

    public static void main(String[] args) {
        Calculator calc = new Calculator();
        // test add
        if ( calc.add(3,4)!=7 ) { System.err.println("error: add(3,4)"); }
        if ( calc.add(0,4)!=4 ) { System.err.println("error: add(0,4)"); }
        ...
    }
}

You can run your test very simply by just running that class:

$ java Calculator

The problem is that your test code will get deployed with the rest of your code. This is inefficient so you could just cut-n-paste this code into a separate class:

class TestCalculator {
    public static void main(String[] args) {
        Calculator calc = new Calculator();
        // test add
        if ( calc.add(3,4)!=7 ) { System.err.println("error: add(3,4)"); }
        if ( calc.add(0,4)!=4 ) { System.err.println("error: add(0,4)"); }
        ...
    }
}

This leaves Calculator clean and properly encapsulated. Now you would run your tests via:

$ java TestCalculator

These tests properly identify which test has failed (if any), but the messages are not great and the IFs make the test code hard to read. Factoring out the test harness code is a good idea:

class TestCalculator {
    public static void main(String[] args) {
        Calculator calc = new Calculator();
        // test add
        assertEquals(calc.add(3,4),7,"add(3,4)");
        assertEquals(calc.add(0,4),4,"add(0,4)");
        ...
    }

    public static void assertEquals(int result, int expecting, String msg) {
        if ( result!=expecting ) {
            System.err.println("failure: "+msg+"="+result+";
                expecting="+expecting);
        }
    }
}

Now, when a test fails, you'll see something like

failure: add(3,4)=932; expecting=7

What if you want multiple tests? You could just include them in the main:

class TestCalculator {
    public static void main(String[] args) {
        Calculator calc = new Calculator();
        // test add
        assertEquals(calc.add(3,4),7,"add(3,4)");
        assertEquals(calc.add(0,4),4,"add(0,4)");
        ...

        // test mult
        assertEquals(calc.mult(0,4),0,"mult(0,4)");
        ...
    }

    public static void assertEquals(int result, int expecting, String msg) {
        ...
    }
}

The problem is that you will get a really big main method and you have to comment out code to test a subset of them. You can extract methods, cleaning up the class a bit also:

class TestCalculator {
  Calculator calc;

  public void testAdd() {
    assertEquals(calc.add(3,4),7,"add(3,4)");
    assertEquals(calc.add(0,4),4,"add(0,4)");
  }

  public void testMult() {
    assertEquals(calc.mult(0,4),0,"mult(0,4)");
  }

  public static void main(String[] args) {
    TestCalculator tester = new TestCalculator();
    tester.setUp();
    tester.testAdd();
    tester.testMult();
    tester.tearDown();
  }

  // support code

  public void assertEquals(int result, int expecting, String msg) {
    ...
  }

  public void setUp() { calc = new Calculator(); }
  public void tearDown() { calc=null; }
}

In fact, you can pass in a method name to test as an argument:

class TestCalculator {
  ...

  public static void main(String[] args) {
    TestCalculator tester = new TestCalculator();
    tester.setUp();
    if ( args[0].equals("add") ) {
      testAdd();
    }
    ...
  }
}

Runnning a test is done via:

$ java TestCalculator add

Congratulations, you just re-invented what jUnit automates for you.

Getting Started

First, make sure that your CLASSPATH has /home/public/cs601/junit.jar in it so that your test suites can see jUnit.

Then, all you have to do is subclass TestCase and make a method starting with test. jUnit will automatically (using reflection) find these methods. Here is the Calculator class by itself, which you will be testing with your test suite:

public class Calculator {
    public int add(int x, int y) { return x+y; }
    public int mult(int x, int y) { return x*y; }
}

Here is a simple TestCalculator that uses jUnit instead of your own testing facility:

import junit.framework.TestCase;

public class TestCalculator extends TestCase {
    Calculator calc;
    public void testAdd() {
        assertEquals("testing add(3,4)", 7, calc.add(3,4));
    }
    public void setUp() { calc = new Calculator(); }
}

To run the test, just compile and invoke the TestRunner:

$ javac TestCalculator.java
$ java junit.textui.TestRunner TestCalculator
.
Time: 0.02

OK (1 test)

Or, using the GUI tool via

$ java junit.swingui.TestRunner TestCalculator

If you click the "..." button next to the class name textfield, junit will bring up all classes in your CLASSPATH that have Test in it or that satisfy the junit interface (it knows the difference).

You can configure this tool to reload your classes from the CLASSPATH every time so that you do not have to exit the tool to change test code.

Failed Tests

What happens when jUnit finds a failed test? To check that out, change the test to have a bad return value:

    public void testAdd() {
        assertEquals("testing add(3,4)", 8, calc.add(3,4));
    }

When you run the test, you will see the failure notice:

$ java junit.textui.TestRunner TestJavaString
.F
Time: 0.003
There was 1 failure:
1) testAdd(TestCalculator)junit.framework.AssertionFailedError:
testing add(3,4) expected:<8> but was:<7>
        at TestCalculator.testAdd(TestCalculator.java:6)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

FAILURES!!!
Tests run: 1,  Failures: 1,  Errors: 0

Running a subset of tests

If you would like run only a subset of a class' testXXX methods, you'll need to provide a suite method that returns the set you want to run. Here I have added a testMult method and added a suite:

import junit.framework.*;

public class TestCalculator extends TestCase {
    Calculator calc;

    public TestCalculator(String name) {
        super(name);
    }

    public void testAdd() {
        assertEquals("testing add(3,4)", 7, calc.add(3,4));
    }
    public void testMult() {
        assertEquals("testing mult(0,4)", 0, calc.mult(0,4));
    }
    public void setUp() { calc = new Calculator(); }

    public static Test suite() {
        TestSuite suite = new TestSuite();
        suite.addTest(new TestCalculator("testAdd"));
        suite.addTest(new TestCalculator("testMult"));
        return suite;
    }
}

You have to have a constructor in this case so that TestCase knows which method (i.e., test) to run.

When you run this now, you will see two tests.

$ java junit.textui.TestRunner TestCalculator
..
Time: 0.003

OK (2 tests)

Using the GUI, you can ask jUnit to rerun any of the tests individually.

Running junit from within code

If you have lots of test suites to run and/or you would like to run junit from a java program instead of the command line, just execute the run method:

public static void main(String[] args) {
  junit.textui.TestRunner.run(TestCalculator.suite());
  // other suites here...
}

In this case, you would just run your class not the junit class:

$ java TestCalculator
..
Time: 0.013

OK (2 tests)