Storing Pylint reports ignores combined_severity_map.json for unspecified labels

[spursbob]

Describe the bug
PR #3414 introduced a pylint.json file. For labels missing in pylint.json the end result is the severity is UNSPECIFIED even if the label and severity is added to combined_severity_map.json.

CodeChecker version
v6.19.1

To Reproduce
Steps to reproduce the behaviour:

Prior to running build and deploy a CodeChecker server with "fixme" severity mapped in combined_severity_map.json set to env var CC_SEVERITY_MAP_FILE

1. Run command 'pylint -f json ${workspace} > pylint_report.json || true' against a workspace with a # TO DO: blah blah which should produce a "fixme" linting output.
2. Run command in CodeChecker '. /install/codechecker/venv/bin/activate'
3. Run command '/install/codechecker/build/CodeChecker/bin/report-converter -t pylint -o ./codechecker_pylint_reports ./pylint_report.json'
4. Run command '/install/codechecker/build/CodeChecker/bin/CodeChecker store ./codechecker_pylint_reports --url http://codechecker.com/Default --name Default
5. "fixme" UNSPECIFIED

Expected behaviour
I would hope to see "fixme" set to the value in combined_severity_map.json

Additional context
I wasn't sure if pylint.py was generating pylist.json on the fly but it seems is looking at the pre-existing pylint.json file which is missing a number of things from "pylint --msg-list" output. My workaround is to modify "codechecker/config/labels/analyzers/pylint.json" in the web server. Perhaps this is all working as expected and it should be a feature request.

[spursbob]

I'm now assuming the CC_SEVERITY_MAP_FILE is no longer an option so now all our eslint serverities are UNSPECIFIED.

[bruntib]

Hi,

CC_SEVERITY_MAP_FILE environment variable was phased out in version 6.17.0. In this version we introduced the labeling feature. Earlier checker severity, profiles and guidelines were listed in separate JSON files. The one containing severity could have been provided by CC_SEVERITY_MAP environment variable. Since 6.17.0 all the severity, profiles, guidelines, links to documentations, and further custom labels can be given in a single JSON file for each analyzer tool. pylint.json is such a file.

We decided that custom JSON files can't be used, so there is no environment variable which overrides the default checker labels. To be precise, there is CC_TEST_LABELS_DIR, but as its name shows, it is created for testing purposes, so it's not recommended to use it. You should add your custom labels to pylint.json directly.

By the way we were thinking on custom label files, but it raised too many complications. Should such a custom file extend or override the default labels provided in config/labels/analyzers directory (both solutions have pros and cons)? If a user provides a custom label file with own severity levels, then how does CodeChecker server get informed about these so they can be displayed in the GUI? Should such a custom severity level be uploaded with CodeChecker store command? What if two different users upload different severity rating for the same checker?

Would you think that this answers your question? Is there some use-case that still needs a solution?

[spursbob]

Thanks, I was able to add more to pylint.jason. However, I'm wondering how I get severity values for Eslint and Spotbugs.

[bruntib]

Unfortunately the labeling JSON files are created manually, so you have to add severity levels manually too. We have some helper scipts for some tools that could be adapted for Eslint and Spotbugs: https://github.com/Ericsson/codechecker/tree/master/scripts/labels.