Add improver pipeline to flag ghost packages #644 #917 #1395

[keshav-space]

This PR introduces the aboutcode pipeline for defining and running improvements on data (see #1395). It also adds a new pipeline to flag ghost packages. Ghost packages are those packages that are not present in the upstream package manager index.

• Resolves Add improver to validate that collected purl are real. #644
• Resolves Add Improver to find out Ghost packages #917
• Refrence VCIO-next: Design new Improver #1395

[pombredanne]

Here are a few nits for your review.

[pombredanne]

Thanks... here a few nits for your review... I wonder if there is a real need to convert versions with univers for this use case.

Also:

• Thanks for adding logging!
• Please add a CHANGELOG entry for logging and pipeline introduction

[pombredanne]

To check for a ghost, why do we need to convert things to a version class?
Could we just check the if the version string exists?
Otherwise we are eventually ghosting all the packages with a version we cannot parse

[keshav-space]

To check for a ghost, why do we need to convert things to a version class?
Could we just check the if the version string exists?

Simple string comparison may lead to incorrectly flagging legitimate versions as ghost packages:

"1.2.0" != "1.02.0" 
"v1.2.3" != "1.2.3"
"1.0" != "1.0.0"

Otherwise we are eventually ghosting all the packages with a version we cannot parse

If we cannot parse the version for a given base_purl, we skip those and don’t mark them as ghosts.

[pombredanne]

Good to go!

[armijnhemel]

Would this also address situations where the upstream index is wrong, such as #915 ?

[keshav-space]

Would this also address situations where the upstream index is wrong, such as #915 ?

@armijnhemel at present, this will flag the ghost packages from the following ecosystems: cargo, composer, conan, deb, gem, github, golang, hex, maven, npm, nuget, and pypi. Once we have support for alpine in https://github.com/aboutcode-org/fetchcode/blob/master/src/fetchcode/package_versions.py, this will also flag ghost packages for the alpine ecosystem.

[armijnhemel]

Would this also address situations where the upstream index is wrong, such as #915 ?

@armijnhemel at present, this will flag the ghost packages from the following ecosystems: cargo, composer, conan, deb, gem, github, golang, hex, maven, npm, nuget, and pypi. Once we have support for alpine in https://github.com/aboutcode-org/fetchcode/blob/master/src/fetchcode/package_versions.py, this will also flag ghost packages for the alpine ecosystem.

I am not sure you understood my question: what if upstream is wrong (as is the case in Alpine see #915 )? Are you always assuming that the index is correct?

[keshav-space]

Would this also address situations where the upstream index is wrong, such as #915 ?

@armijnhemel at present, this will flag the ghost packages from the following ecosystems: cargo, composer, conan, deb, gem, github, golang, hex, maven, npm, nuget, and pypi. Once we have support for alpine in https://github.com/aboutcode-org/fetchcode/blob/master/src/fetchcode/package_versions.py, this will also flag ghost packages for the alpine ecosystem.

I am not sure you understood my question: what if upstream is wrong (as is the case in Alpine see #915 )? Are you always assuming that the index is correct?

@armijnhemel Yes, this pipeline assumes that package index is right. We should have seperate pipeline for case where index is wrong. We will need some generalized way to conclude that package index is reporting wrong version.

[armijnhemel]

Would this also address situations where the upstream index is wrong, such as #915 ?

@armijnhemel at present, this will flag the ghost packages from the following ecosystems: cargo, composer, conan, deb, gem, github, golang, hex, maven, npm, nuget, and pypi. Once we have support for alpine in https://github.com/aboutcode-org/fetchcode/blob/master/src/fetchcode/package_versions.py, this will also flag ghost packages for the alpine ecosystem.

I am not sure you understood my question: what if upstream is wrong (as is the case in Alpine see #915 )? Are you always assuming that the index is correct?

@armijnhemel Yes, this pipeline assumes that package index is right. We should have seperate pipeline for case where index is wrong. We will need some generalized way to conclude that package index is reporting wrong version.

So far I have only seen it once (in Alpine, see #915) so perhaps it could just be added as an exception without building a separate pipeline but I guess it might happen more often.

[pombredanne]

@armijnhemel The general approach to curation is going to be:

1. use federatedcode to store backing data in VCS repos
2. treat a curation outcome as ultimately creating a new "advisory" like record there
3. use an importer that treat this advisories as any other data sources, but with an increased priority, and also being aware of the historical context of this curation.