Provide header level details for a scan: Enhance scancode to include a log or history with useful statistics

[DennisClark]

A run of scancode should generate a log file with meaningful statistics, including such things as:

• the version number of scancode that was executed
• start date/time
• end date/time
• elapsed time
• name of the library that was scanned
• number of files scanned
• {{anything else that makes sense and is useful}}

[balusarakesh]

This log file can also contain the reason for the scan failure if the scan is interrupted in the middle due to any particular reasons.

[pombredanne]

See also aboutcode-org/aboutcode#7

[sschuberth]

As discussed in #840, having a summary of errors (e.g. in the header of the regular JSON output file) would also be beneficial.

[pombredanne]

There is some improvements in develop post #885. More work is needed towards this though

[pombredanne]

GH closed this automatically.... reopening!

[pombredanne]

Here is what we have now:
$ ./scancode -clip -n4 --summary --json-pp j.son samples

{
  "headers": [
    {
      "tool_name": "scancode-toolkit",
      "tool_version": "2.9.7.post183.795fcc4",
      "options": {
        "input": "samples",
        "--copyright": true,
        "--info": true,
        "--json-pp": "j.son",
        "--license": true,
        "--package": true,
        "--processes": "4",
        "--summary": true
      },
      "notice": "Generated with ScanCode and provided on an \"AS IS\" BASIS, WITHOUT WARRANTIES\nOR CONDITIONS OF ANY KIND, either express or implied. No content created from\nScanCode should be considered or used as legal advice. Consult an Attorney\nfor any legal advice.\nScanCode is a free software code scanning tool from nexB Inc. and others.\nVisit https://github.com/nexB/scancode-toolkit/ for support and download.",
      "start_timestamp": "2018-11-29T072242.399469",
      "end_timestamp": "2018-11-29T072250.772025",
      "message": null,
      "errors": [],
      "extra_data": {
        "files_count": 33
      }
    }
  ],
...

and then the same file reprocessed

$ ./scancode --from-json j.son --only-findings --json-pp j2.son --csv j2.csv

{
  "headers": [
    {
      "tool_name": "scancode-toolkit",
      "tool_version": "2.9.7.post183.795fcc4",
      "options": {
        "input": "samples",
        "--copyright": true,
        "--info": true,
        "--json-pp": "j.son",
        "--license": true,
        "--package": true,
        "--processes": "4",
        "--summary": true
      },
      "notice": "Generated with ScanCode and provided on an \"AS IS\" BASIS, WITHOUT WARRANTIES\nOR CONDITIONS OF ANY KIND, either express or implied. No content created from\nScanCode should be considered or used as legal advice. Consult an Attorney\nfor any legal advice.\nScanCode is a free software code scanning tool from nexB Inc. and others.\nVisit https://github.com/nexB/scancode-toolkit/ for support and download.",
      "start_timestamp": "2018-11-29T072242.399469",
      "end_timestamp": "2018-11-29T072250.772025",
      "message": null,
      "errors": [],
      "extra_data": {
        "files_count": 33
      }
    },
    {
      "tool_name": "scancode-toolkit",
      "tool_version": "2.9.7.post183.795fcc4",
      "options": {
        "input": "j.son",
        "--csv": "j2.csv",
        "--from-json": true,
        "--json-pp": "j2.son",
        "--only-findings": true
      },
      "notice": "Generated with ScanCode and provided on an \"AS IS\" BASIS, WITHOUT WARRANTIES\nOR CONDITIONS OF ANY KIND, either express or implied. No content created from\nScanCode should be considered or used as legal advice. Consult an Attorney\nfor any legal advice.\nScanCode is a free software code scanning tool from nexB Inc. and others.\nVisit https://github.com/nexB/scancode-toolkit/ for support and download.",
      "start_timestamp": "2018-11-29T072338.656792",
      "end_timestamp": "2018-11-29T072338.691748",
      "message": null,
      "errors": [],
      "extra_data": {
        "files_count": 0
      }
    }
  ],
...

[pombredanne]

The only question left is about ordering: for now the top most header item is the oldest, not the newest. It might be better to have the ordering done the other way?

[pombredanne]

@sschuberth also we now have a global errors attribute in it too. See this example:
$ ./scancode -clipeu --json-pp - --timeout 0.000001 --verbose tests/scancode/data/failing/patchelf.pdf

{
  "headers": [
    {
      "tool_name": "scancode-toolkit",
      "tool_version": "2.9.7.post183.795fcc4",
      "options": {
        "input": "tests/scancode/data/failing/patchelf.pdf",
        "--copyright": true,
        "--email": true,
        "--info": true,
        "--json-pp": "-",
        "--license": true,
        "--package": true,
        "--timeout": "1e-06",
        "--url": true,
        "--verbose": true
      },
      "notice": "Generated with ScanCode and provided on an \"AS IS\" BASIS, WITHOUT WARRANTIES\nOR CONDITIONS OF ANY KIND, either express or implied. No content created from\nScanCode should be considered or used as legal advice. Consult an Attorney\nfor any legal advice.\nScanCode is a free software code scanning tool from nexB Inc. and others.\nVisit https://github.com/nexB/scancode-toolkit/ for support and download.",
      "start_timestamp": "2018-11-29T073446.380441",
      "end_timestamp": "2018-11-29T073448.617701",
      "message": null,
      "errors": [
        "Path: patchelf.pdf\n  ERROR: for scanner: info:\n  ERROR: Processing interrupted: timeout after 0 seconds.\n  ERROR: for scanner: licenses:\n  ERROR: Processing interrupted: timeout after 0 seconds.\n  ERROR: for scanner: copyrights:\n  ERROR: Processing interrupted: timeout after 0 seconds.\n  ERROR: for scanner: packages:\n  ERROR: Processing interrupted: timeout after 0 seconds.\n  ERROR: for scanner: emails:\n  ERROR: Processing interrupted: timeout after 0 seconds.\n  ERROR: for scanner: urls:\n  ERROR: Processing interrupted: timeout after 0 seconds."
      ],
      "extra_data": {
        "files_count": 1
      }
    }
...

[sschuberth]

The only question left is about ordering

I don't really think it matters, as to be on the safe side you should always sort by start_timestamp anyway. But what about the data following the header? How do you know which data belongs to which header? Or will there always only be data from the last run in the file?

[pombredanne]

@sschuberth re

But what about the data following the header? How do you know which data belongs to which header? Or will there always only be data from the last run in the file?

This is the data as it is from the last run in the file. Tracking actual changes is something to do outside.
Here the headers is just a way to document the fact a file was created through multiple tools touching it, such as multiple scancode runs, editing in aboutcode manager, matching against an index, etc

[sschuberth]

Here the headers is just a way to document the fact a file was created through multiple tools touching it

I see. Another idea to make this more clear would be to always only keep one top-level header, and move headers from previous processing steps e.g. to the existing extra_data field.

[pombredanne]

@sschuberth re

Another idea to make this more clear would be to always only keep one top-level header, and move headers from previous processing steps e.g. to the existing extra_data field.

I am not inclined to go that way: this would mean that each tool that updates the header would need to move several data bits around instead of just appending a whole new record. I would prefer keep this simpler way unless you feel strongly about it

[sschuberth]

I would prefer keep this simpler way unless you feel strongly about it

No, not strongly enough 😉

[pombredanne]

@sschuberth thanks!

[pombredanne]

I am closing at last as this is now merged in develop.
Thank you all for the help and review