Group multiple artifacts of a package version in a set

[pombredanne]

There are several cases where the relationship between two artifacts of the same package version matters such as tracking a binary and its (assumed) source code together and being able to supplement certain package metadata with details from multiple artifacts. It is also essential to get access to the source code to achieve the goals of this project.

This feature should group together multiple package(s) artifact(s) such as multiple builds of the same package version and the source, patch and Git repo archives in a set that can then be used at query time.

It could also be about grouping multiple manifests or data sources (such as an API and a manifest)

Some of the things I envision include

• Enhance the data models to support a package set attribute (some UUID?), (or a relationship between packages)
• Create an API endpoint to return enhanced results for a package set, either by default or as an option
• Develop various procedures to group packages in a set using PURL-based heuristics or else. These could be triggered at package creation time for instance when the source and binary packages are created together, or there could something like the VCIO improvers to the same effect.

[pombredanne]

This is now completed.

• We created a new PackageSet model as seen at https://github.com/nexB/purldb/blob/ed53bcf9dfb1d0ea6e694efa3949556efd465cc7/packagedb/models.py#L1401 and https://github.com/nexB/purldb/blob/ed53bcf9dfb1d0ea6e694efa3949556efd465cc7/packagedb/migrations/0069_packageset_package_package_sets.py

• Model: The new model is fed by various processes such as related packages: v3.0.0...main#diff-db2d9ec22dc66afbcbee16e11efbc61da4118a1f081d44fdc1d662057fa8cd69R352
  and also during package creation for npms and maven packages.

• REST API: there is now an endpoint with enhanced results for a package set. We created an API endpoint and this new endpoint combines together all the details of the packages in a set in combined API response.

  • https://github.com/nexB/purldb/blob/ed53bcf9dfb1d0ea6e694efa3949556efd465cc7/packagedb/api.py#L552
  • https://github.com/nexB/purldb/blob/ed53bcf9dfb1d0ea6e694efa3949556efd465cc7/purldb_project/urls.py#L35
  • ... which fed by this serializer: https://github.com/nexB/purldb/blob/ed53bcf9dfb1d0ea6e694efa3949556efd465cc7/packagedb/serializers.py#L258

• Grouping: we developed a procedure to group packages in a set using PURL-based heuristics. A key outcome is to enhance the details of a binary package with the details of source package
  https://github.com/nexB/purldb/blob/ed53bcf9dfb1d0ea6e694efa3949556efd465cc7/packagedb/api.py#L488

** To test the features of this Task, install PurlDB. Run it locally then:

1. add a package using the new on demand data collection endpoint
   https:///api/collect/?purl=pkg:maven/com.drewnoakes/metadata-extractor@2.19.0
2. browse the package:
   https:///api/packages/?purl=pkg:maven/com.drewnoakes/metadata-extractor@2.19.0

See the set for this package with 2 packages (source and binaries)

3. You can also query the enhanced package details in the set using:
   https:///api/packages//get_enhanced_package_data/ from the details of a package using the “extra data” dropdown.