Reducing the effort of creating JUnit test oracles

So you've gone to the trouble of defining a set of inputs that exercise a particular path through your code. Now you need to write the oracle: the chunk of code that verifies the result is what you expect. This can often be just as much effort as carefully choosing those test inputs. It can also be quite error-prone -- if you're like me, you find a lot more bugs in your tests than in the code your testing!

Here's an approach that can save some time in certain cases. You serialize the result of the test to a text file, then verify (by inspection) that it matches your expectations. For subsequent runs of the test, your test driver reloads the previous result and compares it to the result obtained on the current run.

There are some issues, of course. The test can be a bit more fragile -- if any field of the test result changes, the test will fail. But then updating the oracle will be quick -- just need to dump the new result to a text file, verify it again, then make it the new expected result.  Diff utilities can speed up the re-verification process.

Serialization of the result can get tricky, especially if your result is a tree of objects. Some objects in the tree may not lend themselves to serialization. For others, it may not be appropriate to consider them part of the test result. In cases like this, you may need to dig in to the documentation of the serialization library you are using. The approach I am proposing here might just not be feasible in some cases.

A third concern is that in inspecting the serialized result you could miss problems that you would catch if you manually wrote the oracle. This is subjective, but after using this technique for a few months, I feel that inspection can catch most problems. And, of course, manually constructed test oracles can have bugs that go undetected too!

Here is a JUnit test driver that implements this approach using the Jackson JSON library for serialization and deserialization. Note that with the setup below, only public fields and getters will be checked (but Jackson can be configured to look at private fields as well).

import java.io.IOException;
import java.io.InputStream;
import java.io.StringWriter;
import java.util.Arrays;
import java.util.Collection;
import java.util.List;

import org.codehaus.jackson.JsonFactory;
import org.codehaus.jackson.JsonGenerator;
import org.codehaus.jackson.JsonNode;
import org.codehaus.jackson.JsonParseException;
import org.codehaus.jackson.JsonParser;
import org.codehaus.jackson.map.ObjectMapper;
import org.codehaus.jackson.util.DefaultPrettyPrinter;

import static org.junit.Assert.*;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.junit.runners.Parameterized.Parameters;

@RunWith(Parameterized.class)
public class AutoGeneratedOracleExampleTest {
    private JsonNode testData;
    private String testName;
   
    @Parameters
    public static Collection<Object[]> getParameters() {
        return Arrays.asList(new Object[][]{
                { "Test1" },
                { "Test2" },
                });
    }
   
    public  AutoGeneratedOracleExampleTest (String testName ) 
            throws JsonParseException, IOException {
        this.testName = testName;
    }
   
    @Before
    public void setup() throws JsonParseException, IOException {
        InputStream testDataStream =
            CodeElementExtractorTest.class.getResourceAsStream(
                "AutoGeneratedOracleExampleTestData.json");
        if (testDataStream == null) {
            throw new IOException("Test data file not found");
        }

        try {             
            JsonFactory factory = new JsonFactory();
            JsonParser parser = factory.createJsonParser(testDataStream);
            parser.setCodec(new ObjectMapper());
            testData = parser.readValueAsTree();
        } finally {
            testDataStream.close();
        }
    }
   
    @Test
    public void runTest() throws IOException {
        Object result = ... 
       
        JsonNode expected = getExpectedResult(testName);
        assertEquals("formatted actual: " + getJsonString(result), 
            expected, getJsonNode(result));
    }
   
    private JsonNode getExpectedResult(String testKey) {
        return testData.get(testKey);
    }
   
    private JsonNode getJsonNode(Object obj) throws JsonParseException, IOException {
        JsonFactory factory = new JsonFactory();
        JsonParser parser = factory.createJsonParser(getJsonString(obj));
        parser.setCodec(new ObjectMapper());
        return parser.readValueAsTree();
    }
   
    private String getJsonString(Object obj) throws IOException {
        ObjectMapper mapper = new ObjectMapper();
        StringWriter writer = new StringWriter();
       
        JsonGenerator jsonGenerator =
            mapper.getJsonFactory().createJsonGenerator(writer);
        jsonGenerator.setPrettyPrinter(new DefaultPrettyPrinter());

        mapper.writeValue(jsonGenerator, obj);
        return writer.toString();
    }
}