In the following, we describe how to replicate the evaluation from our paper step-by-step. The instructions explain how to build the Docker image and run the validation in a Docker container.
Start the docker daemon.
# Windows Command Prompt:
- Press 'Windows Key + R' on your keyboard
- Type in 'cmd'
- Click 'OK' or press 'Enter' on your keyboard
# Windows PowerShell:
- Open the search bar (Default: 'Windows Key') and search for 'PowerShell'
- Start the PowerShell
# Linux:
- Press 'ctrl + alt + T' on your keyboard
Clone this repository to a directory of your choice using git:
git clone https://github.com/VariantSync/DiffDetective.git
Then, navigate to the root of your local clone of this repository
cd DiffDetective
and checkout the branch of the replication:
git checkout vamos24_explanations
Finally, navigate to the replication directory
cd replication/VaMoS24-explanations
To build the Docker container you can run the build
script corresponding to your operating system:
# Windows:
.\build.bat
# Linux/Mac (bash):
./build.sh
To execute the replication you can run the execute
script corresponding to your operating system with replication
as first argument.
.\execute.bat replication
./execute.sh replication
WARNING! The replication will at least require an hour and might require up to a day depending on your system. If you want to stop the execution, you can call the provided script for stopping the container in a separate terminal. When restarted, the execution will continue processing by restarting at the last unfinished repository. If you want to replicate parts of the evaluation on a subset of the datasets, run the replication on a custom dataset (see below for instructions).
.\stop-execution.bat
./stop-execution.sh
You might see warnings or errors reported from SLF4J like Failed to load class "org.slf4j.impl.StaticLoggerBinder"
which you can safely ignore.
Further troubleshooting advice can be found at the bottom of this file.
The results of the verification will be stored in the results directory.
The results directory contains a folder relationshipEdges. This folder contains the statistics for every repository, i.e., total run time, run times per commit, fastest/slowest commit, and statistics about the constructed edge-typed variation diffs, and all analysed variation diffs in linegraph format.
You can also run DiffDetective on other datasets by providing the path to the dataset file as first argument to the execution script:
.\execute.bat path\to\custom\dataset.md
./execute.sh path/to/custom/dataset.md
The input file must have the same format as the other dataset files (i.e., repositories are listed in a Markdown table). You can find dataset files in the docs/datasets folder.
Problem:
This is a common problem under Linux, if the user trying to execute Docker commands does not have the permissions to do so.
Fix:
You can fix this problem by either following the post-installation instructions, or by executing the scripts in the replication package with elevated permissions (i.e., sudo
).
Problem:
The Docker container could not be found. This either means that the name of the container that was built does not fit the name of the container that is being executed (this only happens if you changed the provided scripts), or that the Docker container was not built yet.
Fix:
Follow the instructions described above in the section Build the Docker Container
.
No results after verification, or 'cannot create directory '../results/validation/current': Permission denied'
Problem:
This problem can occur due to how permissions are managed inside the Docker container. More specifically, it will appear, if Docker is executed with elevated permissions (i.e., sudo
) and if there is no results directory because it was deleted manually. In this case, Docker will create the directory with elevated permissions, and the Docker user has no permissions to access the directory.
Fix:
If there is a results directory, delete it with elevated permission (e.g., sudo rm -r results
).
Then, create a new results directory without elevated permissions, or execute git restore .
to restore the deleted directory.
Problem:
An operation within the initialization phase of the logger library we use (tinylog) failed.
Fix:
Please ignore this warning. Tinylog will fall back onto a default implementation (Defaulting to no-operation (NOP) logger implementation
) and logging will work as expected.