Initial CycloneDX JSON support #1001

coderpatros · 2021-07-13T11:49:44Z

Demo support for #987

There's heaps missing from this. Just implemented the absolute basics.

A few issues (while they are fresh in my mind):

will have duplicated components across layers
no bom-refs or dependency graph/composition completeness
purl is not correctly generated and only generated for deb packages
files are omitted (although I question the need to include them outside of OSS license compliance use cases, which SPDX output already covers)
demonstrated custom properties with repotags, but plenty of extra tern metadata has not been captured
and probably some other basic stuff that I haven't quite got right in terms of project code consistency, etc

rnjudge · 2021-07-20T14:57:10Z

@coderpatros I saw you joined the zoom call for the community meeting an hour early this morning. Do you have time join now? Apologies for the time confusion, we try to list it in UTC time since it gets confusing with US daylight savings.

rnjudge · 2021-07-22T02:28:13Z

Here's the BOM I get when running with these changes for the photon:3.0 container image. @coderpatros is this about what you would expect?

Details

{
  "bomFormat": "CycloneDX",
  "specVersion": "1.3",
  "serialNumber": "urn:uuid:0d512c96-3f6f-4540-8970-c8d3811b51d1",
  "version": 1,
  "metadata": {
    "timestamp": "2021-07-21T22:37:44Z",
    "tools": [
      {
        "vendor": "Tern Tools",
        "name": "Tern",
        "version": "3181e350efd42f9efabe54b31fa934d95ebe1ee7"
      }
    ],
    "component": {
      "type": "container",
      "scope": "required",
      "name": "photon",
      "version": "fbdae32f534858727fa855af8d548dfa5d98872ef81f466790f7c302a46e8384",
      "hashes": [
        {
          "SHA-256": "fbdae32f534858727fa855af8d548dfa5d98872ef81f466790f7c302a46e8384"
        }
      ],
      "properties": {
        "repotags": [
          "photon:3.0"
        ]
      },
      "components": [
        {
          "type": "container",
          "scope": "required",
          "name": 1,
          "version": "b51a3a7866bca4b8096578a679e302c22afd6dc5d95299b3c17364c9639bb912",
          "hashes": [
            {
              "SHA-256": "b51a3a7866bca4b8096578a679e302c22afd6dc5d95299b3c17364c9639bb912"
            }
          ],
          "properties": {
            "pkg_format": "rpm",
            "os_guess": "VMware Photon OS/Linux"
          }
        }
      ]
    }
  },
  "components": [
    {
      "name": "bash",
      "version": "4.4.18-2.ph3"
    },
    {
      "name": "bzip2-libs",
      "version": "1.0.8-1.ph3"
    },
    {
      "name": "ca-certificates",
      "version": "20190521-1.ph3"
    },
    {
      "name": "ca-certificates-pki",
      "version": "20190521-1.ph3"
    },
    {
      "name": "curl",
      "version": "7.61.1-6.ph3"
    },
    {
      "name": "curl-libs",
      "version": "7.61.1-6.ph3"
    },
    {
      "name": "e2fsprogs-libs",
      "version": "1.45.5-1.ph3"
    },
    {
      "name": "elfutils-libelf",
      "version": "0.176-1.ph3"
    },
    {
      "name": "expat",
      "version": "2.2.9-1.ph3"
    },
    {
      "name": "expat-libs",
      "version": "2.2.9-1.ph3"
    },
    {
      "name": "filesystem",
      "version": "1.1-4.ph3"
    },
    {
      "name": "glibc",
      "version": "2.28-5.ph3"
    },
    {
      "name": "krb5",
      "version": "1.17-1.ph3"
    },
    {
      "name": "libcap",
      "version": "2.25-8.ph3"
    },
    {
      "name": "libdb",
      "version": "5.3.28-2.ph3"
    },
    {
      "name": "libgcc",
      "version": "7.3.0-4.ph3"
    },
    {
      "name": "libmetalink",
      "version": "0.1.3-1.ph3"
    },
    {
      "name": "libsolv",
      "version": "0.6.35-3.ph3"
    },
    {
      "name": "libssh2",
      "version": "1.9.0-2.ph3"
    },
    {
      "name": "lua",
      "version": "5.3.5-2.ph3"
    },
    {
      "name": "ncurses-libs",
      "version": "6.1-2.ph3"
    },
    {
      "name": "nspr",
      "version": "4.21-1.ph3"
    },
    {
      "name": "nss-libs",
      "version": "3.44-3.ph3"
    },
    {
      "name": "openssl",
      "version": "1.0.2u-2.ph3"
    },
    {
      "name": "photon-release",
      "version": "3.0-5.ph3"
    },
    {
      "name": "photon-repos",
      "version": "3.0-5.ph3"
    },
    {
      "name": "popt",
      "version": "1.16-5.ph3"
    },
    {
      "name": "readline",
      "version": "7.0-2.ph3"
    },
    {
      "name": "rpm-libs",
      "version": "4.14.2-7.ph3"
    },
    {
      "name": "sqlite-libs",
      "version": "3.31.1-4.ph3"
    },
    {
      "name": "tdnf",
      "version": "2.1.0-3.ph3"
    },
    {
      "name": "tdnf-cli-libs",
      "version": "2.1.0-3.ph3"
    },
    {
      "name": "toybox",
      "version": "0.8.2-1.ph3"
    },
    {
      "name": "xz-libs",
      "version": "5.2.4-1.ph3"
    },
    {
      "name": "zlib",
      "version": "1.2.11-1.ph3"
    }
  ]
}

coderpatros · 2021-07-22T07:13:53Z

Here's the BOM I get when running with these changes for the photon:3.0 container image. @coderpatros is this about what you would expect?

With the current state of this PR that looks about right. It still needs work and QA on my part.

coderpatros · 2021-07-26T20:36:02Z

@rnjudge @nishakm when you can could please approve the workflow run. I think my local dev environment got a bit messed up. Locally I have failed checks that I've fixed. But I'm not sure I've fixed them all.

coderpatros · 2021-07-26T20:40:06Z

Example output:

{
  "bomFormat": "CycloneDX",
  "specVersion": "1.3",
  "serialNumber": "urn:uuid:5fed8dae-3e13-44a1-903c-1b3fdb98bd18",
  "version": 1,
  "metadata": {
    "timestamp": "2021-07-26T20:14:36Z",
    "tools": [
      {
        "vendor": "Tern Tools",
        "name": "Tern",
        "version": "389aa17f9137eb34c0ff3bc9e0040d7e1d79498f"
      }
    ],
    "component": {
      "type": "container",
      "scope": "required",
      "name": "docker",
      "version": "sha256:a61102937d2bda8319882998ef1ffa27387617f6eea6c298b18a05f7fba82c0d",
      "hashes": [
        {
          "alg": "SHA-256",
          "content": "a61102937d2bda8319882998ef1ffa27387617f6eea6c298b18a05f7fba82c0d"
        }
      ],
      "properties": [
        {
          "name": "tern:repotag",
          "value": "docker:latest"
        },
        {
          "name": "tern:os_guess",
          "value": "Alpine Linux v3.13"
        }
      ],
      "purl": "pkg:docker/docker@sha256:a61102937d2bda8319882998ef1ffa27387617f6eea6c298b18a05f7fba82c0d"
    }
  },
  "components": [
    {
      "name": "musl",
      "version": "1.2.2-r0",
      "type": "application",
      "purl": "pkg:apk/alpine/musl@1.2.2-r0",
      "licenses": [
        {
          "license": {
            "name": "MIT"
          }
        }
      ]
    },
    {
      "name": "busybox",
      "version": "1.32.1-r6",
      "type": "application",
      "purl": "pkg:apk/alpine/busybox@1.32.1-r6",
      "licenses": [
        {
          "license": {
            "name": "GPL-2.0-only"
          }
        }
      ]
    },
    {
      "name": "alpine-baselayout",
      "version": "3.2.0-r8",
      "type": "application",
      "purl": "pkg:apk/alpine/alpine-baselayout@3.2.0-r8",
      "licenses": [
        {
          "license": {
            "name": "GPL-2.0-only"
          }
        }
      ]
    },
    {
      "name": "alpine-keys",
      "version": "2.2-r0",
      "type": "application",
      "purl": "pkg:apk/alpine/alpine-keys@2.2-r0",
      "licenses": [
        {
          "license": {
            "name": "MIT"
          }
        }
      ]
    },
    {
      "name": "libcrypto1.1",
      "version": "1.1.1k-r0",
      "type": "application",
      "purl": "pkg:apk/alpine/libcrypto1.1@1.1.1k-r0",
      "licenses": [
        {
          "license": {
            "name": "OpenSSL"
          }
        }
      ]
    },
    {
      "name": "libssl1.1",
      "version": "1.1.1k-r0",
      "type": "application",
      "purl": "pkg:apk/alpine/libssl1.1@1.1.1k-r0",
      "licenses": [
        {
          "license": {
            "name": "OpenSSL"
          }
        }
      ]
    },
    {
      "name": "ca-certificates-bundle",
      "version": "20191127-r5",
      "type": "application",
      "purl": "pkg:apk/alpine/ca-certificates-bundle@20191127-r5",
      "licenses": [
        {
          "license": {
            "name": "MPL-2.0 AND MIT"
          }
        }
      ]
    },
    {
      "name": "libtls-standalone",
      "version": "2.9.1-r1",
      "type": "application",
      "purl": "pkg:apk/alpine/libtls-standalone@2.9.1-r1",
      "licenses": [
        {
          "license": {
            "name": "ISC"
          }
        }
      ]
    },
    {
      "name": "ssl_client",
      "version": "1.32.1-r6",
      "type": "application",
      "purl": "pkg:apk/alpine/ssl_client@1.32.1-r6",
      "licenses": [
        {
          "license": {
            "name": "GPL-2.0-only"
          }
        }
      ]
    },
    {
      "name": "zlib",
      "version": "1.2.11-r3",
      "type": "application",
      "purl": "pkg:apk/alpine/zlib@1.2.11-r3",
      "licenses": [
        {
          "license": {
            "name": "Zlib"
          }
        }
      ]
    },
    {
      "name": "apk-tools",
      "version": "2.12.5-r0",
      "type": "application",
      "purl": "pkg:apk/alpine/apk-tools@2.12.5-r0",
      "licenses": [
        {
          "license": {
            "name": "GPL-2.0-only"
          }
        }
      ]
    },
    {
      "name": "scanelf",
      "version": "1.2.8-r0",
      "type": "application",
      "purl": "pkg:apk/alpine/scanelf@1.2.8-r0",
      "licenses": [
        {
          "license": {
            "name": "GPL-2.0-only"
          }
        }
      ]
    },
    {
      "name": "musl-utils",
      "version": "1.2.2-r0",
      "type": "application",
      "purl": "pkg:apk/alpine/musl-utils@1.2.2-r0",
      "licenses": [
        {
          "license": {
            "name": "MIT BSD GPL2+"
          }
        }
      ]
    },
    {
      "name": "libc-utils",
      "version": "0.7.2-r3",
      "type": "application",
      "purl": "pkg:apk/alpine/libc-utils@0.7.2-r3",
      "licenses": [
        {
          "license": {
            "name": "BSD-2-Clause AND BSD-3-Clause"
          }
        }
      ]
    },
    {
      "name": "ca-certificates",
      "version": "20191127-r5",
      "type": "application",
      "purl": "pkg:apk/alpine/ca-certificates@20191127-r5",
      "licenses": [
        {
          "license": {
            "name": "MPL-2.0 AND MIT"
          }
        }
      ]
    },
    {
      "name": "openssh-keygen",
      "version": "8.4_p1-r3",
      "type": "application",
      "purl": "pkg:apk/alpine/openssh-keygen@8.4_p1-r3",
      "licenses": [
        {
          "license": {
            "name": "BSD"
          }
        }
      ]
    },
    {
      "name": "ncurses-terminfo-base",
      "version": "6.2_p20210109-r0",
      "type": "application",
      "purl": "pkg:apk/alpine/ncurses-terminfo-base@6.2_p20210109-r0",
      "licenses": [
        {
          "license": {
            "name": "MIT"
          }
        }
      ]
    },
    {
      "name": "ncurses-libs",
      "version": "6.2_p20210109-r0",
      "type": "application",
      "purl": "pkg:apk/alpine/ncurses-libs@6.2_p20210109-r0",
      "licenses": [
        {
          "license": {
            "name": "MIT"
          }
        }
      ]
    },
    {
      "name": "libedit",
      "version": "20191231.3.1-r1",
      "type": "application",
      "purl": "pkg:apk/alpine/libedit@20191231.3.1-r1",
      "licenses": [
        {
          "license": {
            "name": "BSD-3-Clause"
          }
        }
      ]
    },
    {
      "name": "openssh-client",
      "version": "8.4_p1-r3",
      "type": "application",
      "purl": "pkg:apk/alpine/openssh-client@8.4_p1-r3",
      "licenses": [
        {
          "license": {
            "name": "BSD"
          }
        }
      ]
    }
  ]
}

coderpatros · 2021-07-26T20:41:58Z

Output can be validated using either the CycloneDX CLI or Web Tool.

coderpatros · 2021-07-26T20:43:33Z

@sameer1046 I've taken a minimalist approach to this initial support. Including the information I am most interested in. But I'd appreciate your feedback on what is/is not included in the above example output.

sameer1046 · 2021-07-27T05:43:52Z

At this moment the BOM looks nice. Thank you very much.
In future we would like to have more details like the SCM and others details in the BOM for debian packages

nishakm · 2021-07-27T14:47:28Z

@coderpatros Thanks for your work on this! I was on vacation last week. I will review the PR this week.

nishakm

@coderpatros This mostly looks good to me! I have a few questions. Also, if you could add a body to commit 2 in your commit series, that would be awesome :)

Thank you so much for this work!

nishakm · 2021-07-29T14:02:22Z

tern/formats/cyclonedx/cyclonedx.py

@@ -0,0 +1,24 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2019-2020 VMware, Inc. All Rights Reserved.


nishakm · 2021-07-29T14:04:28Z

tern/formats/cyclonedx/cyclonedx.py

+from tern.classes.template import Template
+
+
+class CycloneDX(Template):


The template is meant to provide a mapping from Tern's metadata field names to CycloneDX's field names. It looks like whatever property names Tern uses, CycloneDX uses (name and version). So you don't need this file.

nishakm · 2021-07-29T14:09:50Z

tern/formats/cyclonedx/cyclonedx_common.py

+    'pip': 'pip',
+    'gem': 'gem',
+    'npm': 'npm',
+    'go.mod': 'go',


If go.mod is the only one that is different for purl, you can change the pkg_format in tern/analyze/default/command_lib/base.yml:

go: pkg_format: 'go.mod' <-- change this to 'go'

Then you will not need this mapping.

nishakm · 2021-07-29T14:16:59Z

tern/formats/cyclonedx/cyclonedx_common.py

+
+
+# map Tern OS guesses to package URL namespace
+os_guess_purl_namespace_mapping = {


I wonder if there is an easier way to figure out a valid global namespace for all the ecosystems. Does CycloneDX require a specific string format for namespace or can we just do something like replace spaces with underscores?

This is determined by the package URL spec. This might have been me over thinking future OS support. I'll refactor this part and we can always come back in the future and implement an override mapping for any exceptions instead.

nishakm · 2021-07-29T14:19:17Z

tern/formats/cyclonedx/cyclonedx_common.py

+
+
+def get_os_guess(image_obj):
+    for layer in image_obj.layers:


You can just return image_obj.layers[0].os_guess if it's not empty.

I thought that might have been the case. Thanks

nishakm · 2021-07-29T14:20:12Z

tern/formats/cyclonedx/cyclonedxjson/__init__.py

@@ -0,0 +1,4 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2021 VMware, Inc. All Rights Reserved.


You can use your own Copyright here.

nishakm · 2021-07-29T14:21:25Z

tern/formats/cyclonedx/cyclonedxjson/generator.py

+logger = logging.getLogger(constants.logger_name)
+
+
+def get_document_dict(image_obj_list, template):  # pylint: disable=[unused-argument]


If you're not using template you can remove it.

nishakm · 2021-07-29T14:33:11Z

@coderpatros One more thing: Last tern community meeting notes mention that Rose was concerned about long term maintenance of this format. Do you have any suggestions on how the Tern community can get support for any issues that come up? We do the same thing for SPDX and both the extensions.

coderpatros · 2021-07-29T21:31:47Z

@coderpatros One more thing: Last tern community meeting notes mention that Rose was concerned about long term maintenance of this format. Do you have any suggestions on how the Tern community can get support for any issues that come up? We do the same thing for SPDX and both the extensions.

Backwards compatibility is very important to us. So there shouldn't be much maintenance created by new versions of the spec, unless you want to take advantage of a new capability. I think the more likely cause of maintenance overhead is if you want to refactor Tern or change the Tern data model. That's one of the reasons I've kept this as a minimal implementation. So it only has core information, and in the future, information requested/added by Tern users.

The CycloneDX Slack is probably the best place to ask any general questions. But there's also a mailing list.

But if you get any users raise CycloneDX related issues I'm also happy for you to tag me in the issue to provide feedback.

Also, if you could add a body to commit 2 in your commit series, that would be awesome

After I make the above changes I could push this back up as a single commit if you like? I didn't want to change the existing commit with the README one in case you were already part way through reviewing it.

nishakm · 2021-07-29T22:16:23Z

Backwards compatibility is very important to us. So there shouldn't be much maintenance created by new versions of the spec, unless you want to take advantage of a new capability. I think the more likely cause of maintenance overhead is if you want to refactor Tern or change the Tern data model. That's one of the reasons I've kept this as a minimal implementation. So it only has core information, and in the future, information requested/added by Tern users.

Will you be able to provide a list of missing fields? We could include them in Tern's data model.

The CycloneDX Slack is probably the best place to ask any general questions. But there's also a mailing list.

But if you get any users raise CycloneDX related issues I'm also happy for you to tag me in the issue to provide feedback.

Thank you!

After I make the above changes I could push this back up as a single commit if you like? I didn't want to change the existing commit with the README one in case you were already part way through reviewing it.

The README looks great! You can create one commit :).

coderpatros · 2021-07-31T10:18:52Z

Will you be able to provide a list of missing fields? We could include them in Tern's data model.

A lot of them are hard to populate outside of the build process. But if there is something that I think would be useful I'll raise separate issues for them.

coderpatros · 2021-07-31T10:38:29Z

@nishakm I've made some changes. I've kept them as a separate commit to make it easier to review. But if you're happy with them I'll squash all the commits into a single commit with a more complete commit message before you merge them.

nishakm

Looks good! I'd appreciate it if you could squash your changes into one commit. Sorry for getting back to you so late!

nishakm · 2021-08-04T20:56:01Z

tern/analyze/default/command_lib/base.yml

@@ -434,7 +434,7 @@ npm:
    delimiter: "LICF"
 # golang----------------------------------------------------------------------
 go:
-  pkg_format: 'go.mod'
+  pkg_format: 'go'


- support single or multiple images in output - include basic component information (name, version, hashes, license) - include license evidence - report components by image - add packageurl-python as a dependency - change `go.mod` pkg_format to `go` to align to Package URL spec - include repotag and os_guess as custom properties - update README Signed-off-by: Patrick Dwyer <patrick.dwyer@owasp.org>

coderpatros · 2021-08-04T21:39:56Z

Sorry for getting back to you so late!

No worries @nishakm, we've all got other things going on in our lives.

Should be good to go now.

nishakm · 2021-08-04T23:28:21Z

@coderpatros Looks like something's broken on main. Let me merge in a fix and get back to this.

coderpatros force-pushed the cdx branch 2 times, most recently from 389aa17 to 564257a Compare July 26, 2021 20:34

coderpatros force-pushed the cdx branch from 564257a to a09a4ed Compare July 26, 2021 20:39

coderpatros force-pushed the cdx branch from a09a4ed to 0a6a132 Compare July 26, 2021 20:51

coderpatros changed the title ~~Initial prototype CycloneDX JSON support~~ Initial CycloneDX JSON support Jul 27, 2021

coderpatros marked this pull request as ready for review July 27, 2021 07:21

nishakm requested changes Jul 29, 2021

View reviewed changes

coderpatros requested a review from nishakm July 31, 2021 10:38

nishakm approved these changes Aug 4, 2021

View reviewed changes

coderpatros force-pushed the cdx branch from ce1c04d to ff5aecd Compare August 4, 2021 21:38

nishakm merged commit 9bbb5dd into tern-tools:main Aug 5, 2021

nishakm mentioned this pull request Aug 5, 2021

Support for cyclone dx bom format #987

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial CycloneDX JSON support #1001

Initial CycloneDX JSON support #1001

coderpatros commented Jul 13, 2021

rnjudge commented Jul 20, 2021 •

edited

Loading

rnjudge commented Jul 22, 2021

coderpatros commented Jul 22, 2021

coderpatros commented Jul 26, 2021 •

edited

Loading

coderpatros commented Jul 26, 2021

coderpatros commented Jul 26, 2021

coderpatros commented Jul 26, 2021

sameer1046 commented Jul 27, 2021

nishakm commented Jul 27, 2021

nishakm left a comment

nishakm Jul 29, 2021

nishakm Jul 29, 2021

nishakm Jul 29, 2021

nishakm Jul 29, 2021

coderpatros Jul 29, 2021

nishakm Jul 29, 2021

coderpatros Jul 29, 2021

nishakm Jul 29, 2021

nishakm Jul 29, 2021

nishakm commented Jul 29, 2021

coderpatros commented Jul 29, 2021

nishakm commented Jul 29, 2021

coderpatros commented Jul 31, 2021

coderpatros commented Jul 31, 2021 •

edited

Loading

nishakm left a comment

nishakm Aug 4, 2021

coderpatros commented Aug 4, 2021

nishakm commented Aug 4, 2021

		from tern.classes.template import Template


		class CycloneDX(Template):



		# map Tern OS guesses to package URL namespace
		os_guess_purl_namespace_mapping = {

		logger = logging.getLogger(constants.logger_name)


		def get_document_dict(image_obj_list, template): # pylint: disable=[unused-argument]

Initial CycloneDX JSON support #1001

Initial CycloneDX JSON support #1001

Conversation

coderpatros commented Jul 13, 2021

rnjudge commented Jul 20, 2021 • edited Loading

rnjudge commented Jul 22, 2021

coderpatros commented Jul 22, 2021

coderpatros commented Jul 26, 2021 • edited Loading

coderpatros commented Jul 26, 2021

coderpatros commented Jul 26, 2021

coderpatros commented Jul 26, 2021

sameer1046 commented Jul 27, 2021

nishakm commented Jul 27, 2021

nishakm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nishakm commented Jul 29, 2021

coderpatros commented Jul 29, 2021

nishakm commented Jul 29, 2021

coderpatros commented Jul 31, 2021

coderpatros commented Jul 31, 2021 • edited Loading

nishakm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coderpatros commented Aug 4, 2021

nishakm commented Aug 4, 2021

rnjudge commented Jul 20, 2021 •

edited

Loading

coderpatros commented Jul 26, 2021 •

edited

Loading

coderpatros commented Jul 31, 2021 •

edited

Loading