JUnit and OSGi Test Automation

Overview

While test code has existed for much of the Java/OSGi code, it has remained isolated from the build system, and does not get run regularly. As a result the test code becomes out of date and unused. Since moving to Jenkins, it is easier to setup automated tests and have them visible from the build-output. JUnit tests and OSGi tests have been added to the oe-buglabs-sw2.5 branch of oe-buglabs and Jenkins job.

There are two basic types of tests that are supported, junit and osgi. JUnit tests are simply tests that can be executed from a plain Java main(). JUnit tests are good for testing code that does not require or rely on an OSGi context. An example JUnit test is to verify that the XML parser works. OSGi tests are for tests that require an OSGi service, a bundle to be running, or access to the OSGi framework. An example OSGi test would be to ensure that the HTTP Service is able to host a servlet and that the getPost() method returns the expected data.

Entry-point

The script oe-buglabs/buglabs-scripts/bug-osgi-test.sh in the oe-buglabs repository is run after a successful build is created. The script relies on bitbake to populate the TMPDIR/sysroots with the packaged Java libraries. These libraries are used rather than building them specifically for the tests to ensure that the tests execute what is being shipped.

The script requires no parameters and assumes it's being executed from the root of the workspace. So, from the current directory, a subdir oe-buglabs should exist. All output from the script is generated in a /tests directory from the root

A high-level overview of what the script does

1. validate the environment (current directory)
2. create necessary directories
3. copy binaries from oe build
4. check out and build supplementary code used to run the tests
5. run junit tests
6. run osgi tests

How to add JUnits

To add a JUnit test, it should be in a project separate from the code it is testing. This is done to ensure that the test code is not shipped in the production image.

The ant build file for the test project should include the 'report.src' property which tells the common test code where to put the reports as they are executed:

<property name="report.src" location="." />

The build file should also include the common-osgi.xml ant file which supplies the general code for test execution:

<property name="common.ant.file" location="${base.build.dir}/toolbox/common-osgi.xml" />

And finally, the "test" target for the build file needs to be called from the bug-osgi-test.sh script so the tests are executed as part of the automated build:

ant -Dreport.dir=$TEST_ROOT/junit-reports -Dbase.build.dir=$BUILD_TOOLS -Dcheckout.dir=$TEST_ROOT/bug-osgi -DexternalDirectory=$DEPS_DIR -DdistDirectory=$DEPS_DIR -f $TEST_ROOT/bug-osgi/com.buglabs.common.tests/build.xml test

Validation

After adding these configuration elements, run a fresh build. In the console output you should see something like this that verifies the tests are being executed:

test:
    [junit] Running com.buglabs.common.tests.junit.ClassAvailableTests
    [junit] Tests run: 7, Failures: 0, Errors: 0, Time elapsed: 0.176 sec
    [junit] Running com.buglabs.common.tests.junit.StringSplittingTests
    [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 0.141 sec
    [junit] Running com.buglabs.common.tests.junit.XMLParseTests
    [junit] Tests run: 8, Failures: 0, Errors: 0, Time elapsed: 0.146 sec

How to add OSGi Tests

Since there is no native support for OSGi tests in Ant, a bundle designed for this purpose was written; com.buglabs.osgi.tester. It relies on the KISS principle in that it requires no configuration files to run. Essentially after the bundle is started, it scans the OSGi service registry for any junit.framework.TestSuite services. For each service it loads and executes a TestRunner. It takes the results and generates an XML document that Jenkins can read.

To create an OSGi test, follow the steps for the JUnit tests. You should have a class that extends TestCase. From this point you need to write a BundleActivator instance that will register your TestCase in the OSGi service registry:

regs.add(context.registerService(TestSuite.class.getName(), new TestSuite(OSGiTestCommon.class), null));

This will allow the tests to be executed by com.buglabs.osgi.tester in an automated fashion.

In the ant build file associated with the project that has the tests, a property "testBundles" must be defined which includes a space-delimited list of bundles that will be installed and started before your tests are executed:

<!-- OSGi tests -->
	<property name="testBundles" value="
		file:${distDirectory}/felix-configadmin.jar
		file:${distDirectory}/junit-osgi-4.9b2.jar
		file:${distDirectory}/com.buglabs.osgi.tester.jar
		file:${distDirectory}/com.buglabs.common.jar
		file:${distDirectory}/com.buglabs.common.tests.jar
		">
	</property>

This list should include all (but no more than) the dependencies of your tests and the target bundles of the tests. junit-osgi-4.9b2.jar is added by the automation script. The rest either must come from the shipping rootfs or be built in the automation script.

Finally the tests must be called from the automation script for Jenkins to automatically run them. To this call the "test-osgi" target in the build file associated with your test. In this example, first the tests are compiled and then the tests are executed:

if [ ! -f $DEPS_DIR/com.buglabs.osgi.tester.jar ]; then
	echo "Building OSGi test runner"
	ant -Dbase.build.dir=$BUILD_TOOLS -Dcheckout.dir=$TEST_ROOT/bug-osgi -DexternalDirectory=$DEPS_DIR -DdistDirectory=$DEPS_DIR -f $TEST_ROOT/bug-osgi/com.buglabs.osgi.tester/build.xml build.jars
fi

ant -Dreport.dir=$TEST_ROOT/junit-reports -Dbase.build.dir=$BUILD_TOOLS -Dcheckout.dir=$TEST_ROOT/bug-osgi -DexternalDirectory=$DEPS_DIR -DdistDirectory=$DEPS_DIR -f $TEST_ROOT/bug-osgi/com.buglabs.common.tests/build.xml test-osgi

Validation

If the tests executed you should see output like:

test-osgi:
     [echo] Creating OSGi instance with following bundles:    file:/opt/jenkins/jobs/oe-buglabs-sw2.5/workspace/tests/subjects/felix-configadmin.jar   file:/opt/jenkins/jobs/oe-buglabs-sw2.5/workspace/tests/subjects/junit-osgi-4.9b2.jar   file:/opt/jenkins/jobs/oe-buglabs-sw2.5/workspace/tests/subjects/com.buglabs.osgi.tester.jar   file:/opt/jenkins/jobs/oe-buglabs-sw2.5/workspace/tests/subjects/com.buglabs.common.jar   file:/opt/jenkins/jobs/oe-buglabs-sw2.5/workspace/tests/subjects/com.buglabs.common.tests.jar   
     [java] No config.properties found.
     [java] Test report output directory: /opt/jenkins/jobs/oe-buglabs-sw2.5/workspace/tests/junit-reports
     [java] Waiting 5000 millis for OSGi instance to settle...
     [java] com.buglabs.common.tests.Activator added 1 suites for OSGi testing.
     [java] Running Test Suite: com.buglabs.common.tests.osgi.OSGiTestCommon
     [java] Test Suite Complete: com.buglabs.common.tests.osgi.OSGiTestCommon  Results ~ Errors: 0 Failures: 0

BUILD SUCCESSFUL