reuse lint in CI could produce junit.xml for errors

[ppaalanen]

I started testing reuse lint in a tiny personal project of mine, with CI set up on gitlab.freedesktop.org. I've set the CI to fail when reuse lint fails. However, when it does fail, people will only see that a job failed, but to see why and how, they need to go look at the job log output which is quite a bit "under the hood" experience.

If reuse lint could produce a junit.xml file when the check fails, the details would be much easier to see in the Gitlab UI.

Freedesktop.org CI-templates has a precedent: ci-fairy will produce a junit.xml file when checks fail, which gives the user a nice message in the Gitlab MR front page about what failed. See https://freedesktop.pages.freedesktop.org/ci-templates/ci-fairy.html#checking-merge-requests-within-a-detached-pipeline for how check-merge-request.xml is set up.

This is how it looks like in Gitlab after clicking for failure details:

For reuse lint, I think the detailed message could include the list of non-conformant files.

[carmenbianca]

I think this is very adjacent to #183.

@ppaalanen Question because I'm not super familiar with junit: is junit.xml also commonly used in non-Java domains?

[ppaalanen]

@ppaalanen Question because I'm not super familiar with junit: is junit.xml also commonly used in non-Java domains?

Yes, it is. Gitlab can take junit.xml files and not much else, so all kinds of projects hosted in Gitlab instances have an interest in producing junit.xml files from their test suites. This includes basically everything hosted on https://gitlab.freedesktop.org/, with some notable examples like Mesa (https://www.mesa3d.org/ - FOSS OpenGL, Vulkan, etc. drivers). Mostly because of Gitlab, also https://mesonbuild.com/ supports junit output from test suites.

I think there is some controversy on standardising on junit, but Gitlab is a huge driver for it across everything software, not just Java. See https://gitlab.freedesktop.org/help/ci/unit_test_reports.md .

[alpianon]

More, generally, it would be useful to have a machine-parseable error log (maybe also in json or yaml format?) not necessarily intended to be accessed via GL UI, but also to be passed to an external monitoring platform/dashboard, in order to be able to assess the overall degree of REUSE compliance of a component or of a whole project, and plan actions to improve it

[nicorikken]

Another solution might be the SARIF format, so even particular sections of files can be highlighted. I saw it suggested for CI in another license related tool oss-review-toolkit/ort#1029