Reducing the effort of creating JUnit test oracles

So you've gone to the trouble of defining a set of inputs that exercise a particular path through your code. Now you need to write the oracle: the chunk of code that verifies the result is what you expect. This can often be just as much effort as carefully choosing those test inputs. It can also be quite error-prone -- if you're like me, you find a lot more bugs in your tests than in the code your testing!

Here's an approach that can save some time in certain cases. You serialize the result of the test to a text file, then verify (by inspection) that it matches your expectations. For subsequent runs of the test, your test driver reloads the previous result and compares it to the result obtained on the current run.

There are some issues, of course. The test can be a bit more fragile -- if any field of the test result changes, the test will fail. But then updating the oracle will be quick -- just need to dump the new result to a text file, verify it again, then make it the new expected result.  Diff utilities can speed up the re-verification process.

Serialization of the result can get tricky, especially if your result is a tree of objects. Some objects in the tree may not lend themselves to serialization. For others, it may not be appropriate to consider them part of the test result. In cases like this, you may need to dig in to the documentation of the serialization library you are using. The approach I am proposing here might just not be feasible in some cases.

A third concern is that in inspecting the serialized result you could miss problems that you would catch if you manually wrote the oracle. This is subjective, but after using this technique for a few months, I feel that inspection can catch most problems. And, of course, manually constructed test oracles can have bugs that go undetected too!

Here is a JUnit test driver that implements this approach using the Jackson JSON library for serialization and deserialization. Note that with the setup below, only public fields and getters will be checked (but Jackson can be configured to look at private fields as well).

import java.io.IOException;
import java.io.InputStream;
import java.io.StringWriter;
import java.util.Arrays;
import java.util.Collection;
import java.util.List;

import org.codehaus.jackson.JsonFactory;
import org.codehaus.jackson.JsonGenerator;
import org.codehaus.jackson.JsonNode;
import org.codehaus.jackson.JsonParseException;
import org.codehaus.jackson.JsonParser;
import org.codehaus.jackson.map.ObjectMapper;
import org.codehaus.jackson.util.DefaultPrettyPrinter;

import static org.junit.Assert.*;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.junit.runners.Parameterized.Parameters;

@RunWith(Parameterized.class)
public class AutoGeneratedOracleExampleTest {
    private JsonNode testData;
    private String testName;
   
    @Parameters
    public static Collection<Object[]> getParameters() {
        return Arrays.asList(new Object[][]{
                { "Test1" },
                { "Test2" },
                });
    }
   
    public  AutoGeneratedOracleExampleTest (String testName ) 
            throws JsonParseException, IOException {
        this.testName = testName;
    }
   
    @Before
    public void setup() throws JsonParseException, IOException {
        InputStream testDataStream =
            CodeElementExtractorTest.class.getResourceAsStream(
                "AutoGeneratedOracleExampleTestData.json");
        if (testDataStream == null) {
            throw new IOException("Test data file not found");
        }

        try {             
            JsonFactory factory = new JsonFactory();
            JsonParser parser = factory.createJsonParser(testDataStream);
            parser.setCodec(new ObjectMapper());
            testData = parser.readValueAsTree();
        } finally {
            testDataStream.close();
        }
    }
   
    @Test
    public void runTest() throws IOException {
        Object result = ... 
       
        JsonNode expected = getExpectedResult(testName);
        assertEquals("formatted actual: " + getJsonString(result), 
            expected, getJsonNode(result));
    }
   
    private JsonNode getExpectedResult(String testKey) {
        return testData.get(testKey);
    }
   
    private JsonNode getJsonNode(Object obj) throws JsonParseException, IOException {
        JsonFactory factory = new JsonFactory();
        JsonParser parser = factory.createJsonParser(getJsonString(obj));
        parser.setCodec(new ObjectMapper());
        return parser.readValueAsTree();
    }
   
    private String getJsonString(Object obj) throws IOException {
        ObjectMapper mapper = new ObjectMapper();
        StringWriter writer = new StringWriter();
       
        JsonGenerator jsonGenerator =
            mapper.getJsonFactory().createJsonGenerator(writer);
        jsonGenerator.setPrettyPrinter(new DefaultPrettyPrinter());

        mapper.writeValue(jsonGenerator, obj);
        return writer.toString();
    }
}

Browser extensions that need to call native code

I've been working on a browser extension for Firefox and Chrome that needs to invoke some native code.  It turns out that the only way to do this in Chrome is through an NPAPI plugin (NaCl is not an option here because the native code needs to perform some privileged operations).  Luckily there is the Firebreath project which makes it pretty easy to create a cross-platform NPAPI plugin for Mac, Linux, and Windows.  You can load the plugin once in the background page of your Chrome plugin and then your content scripts can communicate with it through message passing.

In Firefox, however, there are challenges to calling an NPAPI plugin from an extension.  The two options are (a) to load the plugin in to the DOM of the browser itself; or (b) to inject the plugin in to the DOM of each page as it is loaded.  Option (a) seems like a bit of a hack, and (b) has security issues (since scripts on the page could then invoke the plugin).

A better option in Firefox is to use the js-ctypes API to call in to a shared library written in C.  The shared library only needs to be loaded once, and is not accessible to scripts on web pages.  The library is just a vanilla C shared library, you don't have all the overhead of conforming to the NPAPI plugin interface.  The drawback, of course, is that you have to create both an NPAPI plugin and a js-ctypes shared library if you want to support Chrome and Firefox.  But I felt it was worth the extra effort, given the limitations of calling NPAPI plugins from Firefox extensions.

To minimize code duplication, I created a static library which contains most of my functionality, and then linked it into the NPAPI plugin and the js-ctypes shared library.