CI Setup

The current Continous Integration (CI) pipeline uses Github actions to run tests for each push and all pull request. Code coverage information is collected and pushed to coveralls.io.

Overview

We use two workflows, .github/workflows/tests.yml and .github/workflows/vlog-tests.yml to run unit tests and integration tests. VLog integration tests are separated to allow those to fail, while still keeping the build green (e.g., when testing for known upstream bugs in VLog). Both jobs are quite similar, with the integration tests using a needs: unit-tests dependency to ensure they run only when unit tests complete successfully. Essentially, the jobs differ only in the mvn targets they run. Apart from that, they both use the checkout action to obtain the current source code, and the install-nix and cachix-action actions to install the nix package manager, as well as set up the cachix binary cache. Then, before running the tests, information on the commit being built is written to environment variables, so it can be passed on to the coveralls:report goal for submitting the code coverage information. This largely follows this approach. The nix-shell environment that the test are run in is declared in flake.nix, which specifies dependencies on maven, the JDK, and the VLog library, registers the latter with maven, and installs any Java dependencies. A nix flake is used to pin all dependencies to known versions.

Building VLog

VLog (and, particularly, the jvlog.jar library) is built as a Nix package. Over the build-vlog-library.sh script, this offers some advantages: Nix packages specify declaratively how to build a package, and building them is deterministic. This enables easy caching, since as long as the inputs do not change, the cache remains valid. Currently, KOGNAC, Trident, and VLog all require patches to their CMakeLists.txt to work with the provided lz4 libraries from the system.

Testing locally

Having installed Nix and enabled flake support locally, run nix run .#rulewerk to run the rulewerk client, or nix develop to enter a shell environment that has a maven repository with the VLog java library and all dependencies installed.

Caching

Caching is provided by the cachix binary cache, which automatically caches all built Nix artefacts. If you build locally, use cachix use rulewerk-knowledge to substitute cached build artifacts (such as the dependencies), if present.

Coverage

Coverage information is collected by jacoco-maven-plugin, configured in pom.xml. Although unfortunate, the duplicated exclusions appear to be necessary, since otherwise the coveralls-maven-plugin will be unable to find the source code for the generated parser code. Coverage information is aggregated in the coverage module, configured in coverage/pom.xml. To ensure accurate aggregation, all maven modules need to be registered as a dependency of the coverage module. Coverage information is pushed to coveralls.io by the coveralls-maven-plugin.

Updating

Updating VLog, trident, or kognac

VLog, trident, and kognac are pinned to specific git commits in nix/pkgs/vlog/default.nix, nix/pkgs/trident/default.nix, and nix/pkgs/kognac/default.nix, respectively. This is required to achieve a deterministic build. In general, the versions used should be the target we plan to release against: either the latest release, or the most recent commit, if we plan to release in sync with VLog.

To update, replace the rev attribute with the commit hash (you can also use the name of a git tag here), and update the sha256 hash accordingly. You can obtain the correct sha256 hash using nix-prefetch-github --nix karmaresearch <repository> on a NixOS machine. (Alternatively, but not recommended, one can obtain the correct sha256 by updating the rev value, and changing one character in the old sha256 value. The CI build will fail with error: hash mismatch because of the incorrect sha256 value, and the correct hash will be given in the error message of the build, as got: sha256-[correct_sha256].) It might also be necessary to update the files referenced in the patches = [ … ]; line, should the CMakeLists.txt of these projects change.

Updating maven dependencies

Maven dependencies for rulewerk are pinned using the mvn2nix, which writes version information into mvn2nix-lock.json. Should maven dependencies (except for jvlog.jar, which is taken directly from the VLog derivation) change, this mvn2nix-lock.json file will need to be updated. This can be done by running nix run .#mvn2nix. There is also a Github workflow to automatically create pull requests with updated mvn2nix-lock.json on changes to any of the pom.xml files. Make sure to commit the new mvn2nix-lock.json.

Updating other dependencies

Use a flake-enabled nix to run nix flake update. This will update all flake inputs, particularly nixpkgs, which will be used to satisfy system-level dependencies such as zlib, the JDK, and maven. Make sure to commit the new flake.lock.

Updating to a new JDK version

The JDK version is specified using the getJdk function in flake.nix, where let getJdk = pkgs: pkgs.jdk8_headless; uses the headless version of the Java 8 JDK from nixpkgs. To update to, e.g., Java 11, change this line to let getJdk = pkgs: pkgs.jdk11_headless;.

TODOs

While already working, the process can still be improved:

• Although nix is available for, e.g., macOS, the packages have only been tested on Linux and might require adaption to successfully build there.
• It would be nice to add a flake output for the javadoc.
• For building against unstable VLog, it would be nice to have a system to automatically bump the hashes as necessary.