Generate the first SBOM protoype from the Kubernetes release process

[puerco]

What type of PR is this?

/kind feature

What this PR does / why we need it:

This PR integrates the Bill of Materials code Into the release process. This generates the first SBOM prototypes that describe a Kubernetes release. We are generating one SBOM for the source code and one for the release artifacts. The proposed namespaces for these are the following:

• http://k8s.io/sbom/source/v1.22.0-alpha.3
• http://k8s.io/sbom/release/v1.22.0-alpha.3

This PR aims to test the integration of the BOM code with the Kubernetes Release process. The resulting SBOM documents will describe the release source and artifacts, but will not include yet all the features of the SPDX libraries. They will do most of the work of creating the BOM however, enabling us to look for bugs and polish the documents to their final state in beta.0.

This PR is broken into several commits, I've attempted to add meaningful descriptions to each one. The most important changes are:

• Prewarming the license cache during release.PrepareWorkspaceStage() to avoid downloading the license files later
• The release process staging phase now has a GenerateBillOfMaterials() step that builds the SPDX documents.
• We now create an SPDX SBOM describing the Kubernetes source during staging
• Each version in a release now features an SPDX bill of materials listing its binaries and images
• stage.GenerateBillOfMaterials() now has an integration test

Which issue(s) this PR fixes:

Part of: #1837

Special notes for your reviewer:

The SBOM generation has been tested in several test runs of the release process. I've verified that both stage and release work as expected. For details see:

• Mock stage run: https://console.cloud.google.com/cloud-build/builds/c95c4cde-bfa6-470f-ab35-540399f90a0c?project=648026197307
• Mock release run: https://console.cloud.google.com/cloud-build/builds/82f86e12-7a0c-492d-93ef-d1a8b64f9f41?project=648026197307

One of the commits features a temporary function: ListArtifacts(). This function will be replaced with the Supported Platforms framework once we finish it (ref: #1926 )

Does this PR introduce a user-facing change?

* When staging a new kubernetes build, `krel` will now prewarm the license cache to have the classifier data ready when generating the bill of materials.
* The release process staging phase now has a `GenerateBillOfMaterials()` step that builds the SPDX documents.
* We now create an SPDX SBOM describing the Kubernetes source during staging
* Each version in a release now features an SPDX bill of materials listing its binaries and images
* stage.GenerateBillOfMaterials() now has an integration test

[puerco]

/milestone v1.22

[puerco]

I gave it a test run with export TOOL_ORG=puerco && export TOOL_REF=bom-rproc-1 (using a slightly modified version as currently there is a problem with the release process) and it works:

[2021-05-29T06:02:24Z] Found release versions: prime: v1.22.0-alpha.3, alpha: v1.22.0-alpha.3 
INFO[2021-05-29T06:02:24Z] Preparing workspace                           step=5/9
INFO[2021-05-29T06:02:24Z] Caching SPDX license set to /tmp.k8s/spdx/downloadCache 
INFO[2021-05-29T06:02:24Z] Downloading main SPDX license data from https://spdx.org/licenses/ 
INFO[2021-05-29T06:02:24Z] Sending GET request to https://spdx.org/licenses/licenses.json 
INFO[2021-05-29T06:02:25Z] Read data for 460 licenses. Downloading.     
INFO[2021-05-29T06:02:25Z] Downloading license data from https://spdx.org/licenses/UCL-1.0.json 
...
INFO[2021-05-29T06:02:31Z] Downloaded 460 licenses                      
INFO[2021-05-29T06:02:31Z] Preparing workspace for staging in /workspace/src/k8s.io/kubernetes 
INFO[2021-05-29T06:02:31Z] Cloning repository to /workspace/src/k8s.io/kubernetes 

[puerco]

Since this PR has not merged I will further this branch to generate the first BOM prototypes from the release process.
/hold

[puerco]

/test pull-release-integration-test

[puerco]

The SBOM integration is ready to go! 🎉

There are still details to iron out, but the big chunk is done. We are ready to produce the first set of SBOMs for v1.22.0-alpha.3! 🚀

/hold cancel

[dims]

@puerco is the list of things in the SBOM, exactly reflect the stuff in the LICENSES/ directory in k/k? (or does it have more stuff?)

[puerco]

All the data in LICENSES should be included plus a lot more. Most should be there now, but I noticed in the logs that some packages could not be downloaded and hence have no license declared. I have to check why.

This PR lays the groundwork to generate the doc automatically. Once this is in, my focus will be on perfecting the output for beta.0 The licensing data of dependencies will be 1:1 to the LICENSES dir but I have yet to write some scripts to verify.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: puerco, saschagrunert

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/OWNERS [puerco,saschagrunert]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[dims]

@puerco "plus a lot more" sounds like could be a concern...