feat(license): improve license normalization #7131

pbaumard · 2024-07-09T15:04:08Z

Description

Normalize "+", "-only" and "-or-later" suffixes
Many new mappings from oss-review-toolkit
Space normalization, including newlines
Normalize US / UK spelling for license / licence.
Remove "THE " prefix
Remove common suffixes: " LICENSE", " LICENSED", -"LICENSE" and "-LICENSED"
Add MIT-0
Add tests

Related issues

Close feat(license): Improve license normalization #7189

Checklist

I've read the guidelines for contributing to this repository.
I've followed the conventions in the PR title.
I've added tests that prove my fix is effective or that my feature works.
I've updated the documentation with the relevant information (if needed).
I've added usage information (if the PR introduces new options)
I've included a "before" and "after" example to the description (if the PR is a user interface change).

- Remove "THE " prefix - Remove "LICENSE " suffix - Fix lowercase key - Add BSD LICENSE 3 - Add ECLIPSE PUBLIC LICENSE 1.0 - Add MIT-0 - Add tests

CLAassistant · 2024-07-09T15:04:14Z

All committers have signed the CLA.

CLAassistant · 2024-07-09T15:04:14Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

DmitriyLewen

Hello @pbaumard
Thanks for your work!

LGTM.
Can you fix tests?

pkg/licensing/normalize_test.go

pkg/licensing/normalize.go

DmitriyLewen

Hello @pbaumard
Sorry for delay.

Left comments. Take a look when you have time.

pkg/licensing/normalize.go

DmitriyLewen · 2024-07-24T09:48:01Z

pkg/licensing/normalize.go

+var mapping = make(map[string]expression.SimpleExpr)
+
+func addMap(name, key string, hasPlus bool) {
+	license := normalizeKeyAndSuffix(name)


IIUC licenses are already normalized.
Do we need this?

The license from normalizeKeyAndSuffix is only used for the following assertion with the panic error.

I added some comemnts to make it more explicit.

The check could be made in unit test but it means having mapping public.

I think I understand your idea - you want to check the newly added licenses (from another PR later).

I think we don't need to do this check every time we start Trivy.

The test can be done in a unit test, but this means that the mapping will be public.

You have added so many licenses, so I am not sure that many licenses will be added later.
Maybe a comment before mapping to the instructions for the new licenses will be enough. wdyt?

Especially with latest commit with the version regular expression, it's becoming quite difficult to make sure to only add standardized keys.

I added an InvalidMappingKeys used in test and which might also be used later if adding new mappings from CLI or other mean becomes possible.

DmitriyLewen · 2024-07-24T10:11:23Z

pkg/sbom/cyclonedx/marshal.go

@@ -288,7 +290,7 @@ func (*Marshaler) Licenses(licenses []string) *cdx.Licenses {
 	choices := lo.Map(licenses, func(license string, i int) cdx.LicenseChoice {
 		return cdx.LicenseChoice{
 			License: &cdx.License{
-				Name: license,
+				Name: NormalizeLicense(license),


Can we use this function here?

trivy/pkg/licensing/normalize.go

Lines 735 to 737 in 907683d

func Normalize(name string) string {

return NormalizeLicense(name).String()

}

I removed the CycloneDX normalization from this PR to reduce the scope.

From cycloneDX spec that would ideally mean having :

Id: "A valid SPDX license ID"

Name: "If SPDX does not define the license used, this field may be used to provide the license name",

integration/testdata/fluentd-multiple-lockfiles.cdx.json.golden

DmitriyLewen · 2024-07-24T10:36:50Z

pkg/fanal/test/integration/testdata/goldens/packages/amazon-2.json.golden

We do not check all fields for OS packages:

trivy/pkg/fanal/test/integration/library_test.go

Lines 321 to 324 in 051ac39

for i := 0; i < len(expectedPkgs); i++ {

require.Equal(t, expectedPkgs[i].Name, detail.Packages[i].Name, tc.name)

require.Equal(t, expectedPkgs[i].Version, detail.Packages[i].Version, tc.name)

}

Therefore, there is no point in updating the OS packages for the files pkg/fanal/test/integration/testdata/goldens/packages/*.json.golden.

I reverted those changes.

Sorry, I confused you a little.
containerd tests use full files.
I have updated these golden files in 39f796e.

Normalized SPDX license should be in Id field.

DmitriyLewen · 2024-07-25T03:24:40Z

pkg/licensing/normalize.go

+var mapping = make(map[string]expression.SimpleExpr)
+
+func addMap(name, key string, hasPlus bool) {
+	license := normalizeKeyAndSuffix(name)


I think I understand your idea - you want to check the newly added licenses (from another PR later).

I think we don't need to do this check every time we start Trivy.

The test can be done in a unit test, but this means that the mapping will be public.

You have added so many licenses, so I am not sure that many licenses will be added later.
Maybe a comment before mapping to the instructions for the new licenses will be enough. wdyt?

integration/testdata/fluentd-multiple-lockfiles.cdx.json.golden

DmitriyLewen · 2024-07-25T04:33:58Z

pkg/fanal/test/integration/testdata/goldens/packages/amazon-2.json.golden

Sorry, I confused you a little.
containerd tests use full files.
I have updated these golden files in 39f796e.

DmitriyLewen · 2024-07-25T04:34:45Z

pkg/fanal/test/integration/testdata/goldens/packages/vulnimage.json.golden

    },
    "Version": "1.6.3-r0",
    "Arch": "x86_64",
    "SrcName": "apr",
    "SrcVersion": "1.6.3-r0",
    "Licenses": [
-      "ASL2.0"
+      "Apache-2.02.0"


@pbaumard can you take a look?

Fixed in latest commit.

DmitriyLewen · 2024-07-31T08:43:18Z

pkg/licensing/normalize.go

+func addMap(name, key string, hasPlus bool) {
+	mapping[name] = expression.SimpleExpr{License: key, HasPlus: hasPlus}
+}


looks like we don't need this function anymore and we can just put everything in the map right away

Done in latest commit, even if I am not sure it is more readable that way.

DmitriyLewen · 2024-07-31T09:06:52Z

pkg/licensing/normalize.go

+var versionRegexpString = "([A-UW-Z)]{2,})( LICENSE)?\\s*[,(-]?\\s*(V|V\\.|VERSION|VERSION-|-)?\\s*([1-9](\\.\\d)*)[)]?"
+
+// case insensitive version match anywhere in string
+var versionRegexp = regexp.MustCompile("(?i)" + versionRegexpString)
+
+// version suffix match
+var versionSuffixRegexp = regexp.MustCompile(versionRegexpString + "$")


I'm not sure I understand where we need this.
Can you add an example in the comment?

I tried to avoid using regex if possible.
Maybe we can avoid using this regex here

The many version variations are in normalize_test.

Most of the mappings in current trivy version or in oss-review-toolkit/ort are because of slight variations in the way the version is declared in the license.

So this regexp allows to:

greatly limit the number of mappings

avoid missing version mappings

This regexp is strict by checking only version suffixes.

DmitriyLewen · 2024-07-31T09:08:16Z

pkg/licensing/normalize_test.go

@@ -8,6 +8,219 @@ import (
 	"github.com/aquasecurity/trivy/pkg/licensing"
 )

+func TestMap(t *testing.T) {
+	assert.Empty(t, licensing.InvalidMappingKeys((nil)))


Suggested change

assert.Empty(t, licensing.InvalidMappingKeys((nil)))

assert.Empty(t, licensing.InvalidMappingKeys(nil))

InvalidMappingKeys no more used in latest commit.

DmitriyLewen · 2024-07-31T09:11:17Z

pkg/fanal/analyzer/pkg/apk/apk_test.go

@@ -453,3 +450,23 @@ func TestParseApkInfo(t *testing.T) {
 		})
 	}
 }
+
+func TestParseLicense(t *testing.T) {


I think this is a redundant test.
We see this function working in TestParseApkInfo.

TestLaxSplitLicense was moved to normalize_test in latest commit.

DmitriyLewen · 2024-07-31T09:16:34Z

pkg/licensing/normalize.go

+func InvalidMappingKeys(licenseToNormalized map[string]expression.SimpleExpr) []string {
+	if licenseToNormalized == nil {
+		licenseToNormalized = mapping
+	}
+	var invalid []string
+	for key := range licenseToNormalized {
+		standardized := standardizeKeyAndSuffix(key)
+		if standardized.License != key {
+			invalid = append(invalid, key)
+		}
+	}
+	return invalid
+}


What if we move this function to the test.
To get mapping, we can create a function like GetBuiltinRules() for secrets.

normalize_test is now in package licensing to use internal mapping and functions.

…rmalize_test

DmitriyLewen

@pbaumard Thanks for work!
And sorry for delay.

LGTM.
Left 1 small comment.

DmitriyLewen · 2024-08-20T07:44:00Z

pkg/licensing/normalize_test.go

+			[]string{
+				"Apache+",
+			},
+			"Apache-2.0+",
+			"Apache-2.0",


Can you add field names for better readability in these tests?
e.g.:

Suggested change

[]string{

"Apache+",

},

"Apache-2.0+",

"Apache-2.0",

licenses: []string{

"Apache+",

},

normalized: "Apache-2.0+",

normalizedKey: "Apache-2.0",

Field names have been added in last commit.

…e-normalize

DmitriyLewen

Hello @pbaumard
Thanks for your work and sorry for delays.

@knqyf263 take a look, when you have time, please.

Signed-off-by: knqyf263 <knqyf263@gmail.com>

knqyf263

It's an awesome contribution! I planned to add this kind of improvement but didn't find the time. Thanks for your great work and patience.

* feat(vm): Support direct filesystem (aquasecurity#7058) Signed-off-by: yusuke.koyoshi <yusuke.koyoshi@bizreach.co.jp> * feat(cli)!: delete deprecated SBOM flags (aquasecurity#7266) Signed-off-by: knqyf263 <knqyf263@gmail.com> * feat(vm): support the Ext2/Ext3 filesystems (aquasecurity#6983) * fix(plugin): do not call GitHub content API for releases and tags (aquasecurity#7274) Signed-off-by: knqyf263 <knqyf263@gmail.com> * fix(java): Return error when trying to find a remote pom to avoid segfault (aquasecurity#7275) Co-authored-by: DmitriyLewen <dmitriy.lewen@smartforce.io> * fix(flag): incorrect behavior for deprected flag `--clear-cache` (aquasecurity#7281) * refactor(misconf): remove file filtering from parsers (aquasecurity#7289) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * feat(vuln): Add `--detection-priority` flag for accuracy tuning (aquasecurity#7288) Signed-off-by: knqyf263 <knqyf263@gmail.com> * docs: add auto-generated config (aquasecurity#7261) Signed-off-by: knqyf263 <knqyf263@gmail.com> Co-authored-by: knqyf263 <knqyf263@gmail.com> * fix(terraform): add aws_region name to presets (aquasecurity#7184) * perf(misconf): do not convert contents of a YAML file to string (aquasecurity#7292) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * refactor(misconf): remove unused universal scanner (aquasecurity#7293) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * perf(misconf): use json.Valid to check validity of JSON (aquasecurity#7308) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * fix(misconf): load only submodule if it is specified in source (aquasecurity#7112) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * feat(misconf): support for policy and bucket grants (aquasecurity#7284) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * fix(misconf): do not set default value for default_cache_behavior (aquasecurity#7234) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * feat(misconf): iterator argument support for dynamic blocks (aquasecurity#7236) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> Co-authored-by: simar7 <1254783+simar7@users.noreply.github.com> * chore(deps): bump the common group across 1 directory with 7 updates (aquasecurity#7305) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * docs: update client/server docs for misconf and license scanning (aquasecurity#7277) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> Signed-off-by: knqyf263 <knqyf263@gmail.com> Co-authored-by: knqyf263 <knqyf263@gmail.com> * docs: update links to packaging.python.org (aquasecurity#7318) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * perf(misconf): optimize work with context (aquasecurity#6968) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * refactor: replace ftypes.Gradle with packageurl.TypeGradle (aquasecurity#7323) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * docs: update air-gapped docs (aquasecurity#7160) Signed-off-by: knqyf263 <knqyf263@gmail.com> Co-authored-by: knqyf263 <knqyf263@gmail.com> * docs(misconf): Update callsites to use correct naming (aquasecurity#7335) * chore(deps): bump the common group with 9 updates (aquasecurity#7333) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix(misconf): change default TLS values for the Azure storage account (aquasecurity#7345) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * refactor(misconf): highlight only affected rows (aquasecurity#7310) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * fix(misconf): wrap Azure PortRange in iac types (aquasecurity#7357) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * feat(misconf): scanning support for YAML and JSON (aquasecurity#7311) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * feat(misconf): variable support for Terraform Plan (aquasecurity#7228) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * fix: safely check if the directory exists (aquasecurity#7353) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * chore(deps): bump the aws group across 1 directory with 7 updates (aquasecurity#7358) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat(server): add internal `--path-prefix` flag for client/server mode (aquasecurity#7321) Signed-off-by: knqyf263 <knqyf263@gmail.com> * chore(deps): bump trivy-checks (aquasecurity#7350) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * refactor(misconf): use slog (aquasecurity#7295) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * feat(misconf): ignore duplicate checks (aquasecurity#7317) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * fix(misconf): init frameworks before updating them (aquasecurity#7376) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * fix(misconf): support deprecating for Go checks (aquasecurity#7377) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * feat(python): use minimum version for pip packages (aquasecurity#7348) * docs: add pkg flags to config file page (aquasecurity#7370) * feat(misconf): Add support for using spec from on-disk bundle (aquasecurity#7179) * fix(report): escape `Message` field in `asff.tpl` template (aquasecurity#7401) * fix(misconf): use module to log when metadata retrieval fails (aquasecurity#7405) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * feat(misconf): support for ignore by nested attributes (aquasecurity#7205) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * fix(misconf): do not filter Terraform plan JSON by name (aquasecurity#7406) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * feat(misconf): port and protocol support for EC2 networks (aquasecurity#7146) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * chore: fix allow rule of ignoring test files to make it case insensitive (aquasecurity#7415) * fix(secret): use only line with secret for long secret lines (aquasecurity#7412) * chore: update CODEOWNERS (aquasecurity#7398) Signed-off-by: knqyf263 <knqyf263@gmail.com> * feat(server): Make Trivy Server Multiplexer Exported (aquasecurity#7389) * feat(report): export modified findings in JSON (aquasecurity#7383) Signed-off-by: knqyf263 <knqyf263@gmail.com> * fix(sbom): use `NOASSERTION` for licenses fields in SPDX formats (aquasecurity#7403) * fix(misconf): do not register Rego libs in checks registry (aquasecurity#7420) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * chore(deps): Bump trivy-checks (aquasecurity#7417) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> Co-authored-by: nikpivkin <nikita.pivkin@smartforce.io> * fix(misconf): do not recreate filesystem map (aquasecurity#7416) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * fix(secret): use `.eyJ` keyword for JWT secret (aquasecurity#7410) * fix(misconf): fix infer type for null value (aquasecurity#7424) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * fix(aws): handle ECR repositories in different regions (aquasecurity#6217) Signed-off-by: Kevin Conner <kev.conner@getupcloud.com> * fix: logger initialization before flags parsing (aquasecurity#7372) Signed-off-by: knqyf263 <knqyf263@gmail.com> Co-authored-by: knqyf263 <knqyf263@gmail.com> * fix(nodejs): check all `importers` to detect dev deps from pnpm-lock.yaml file (aquasecurity#7387) * test: add integration plugin tests (aquasecurity#7299) * feat(sbom): set User-Agent header on requests to Rekor (aquasecurity#7396) Signed-off-by: Bob Callaway <bcallaway@google.com> * fix(helm): explicitly define `kind` and `apiVersion` of `volumeClaimTemplate` element (aquasecurity#7362) * chore(deps): Bump trivy-checks and pin OPA (aquasecurity#7427) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> Co-authored-by: nikpivkin <nikita.pivkin@smartforce.io> * feat(java): add `test` scope support for `pom.xml` files (aquasecurity#7414) * fix(license): add license handling to JUnit template (aquasecurity#7409) * feat(go): use `toolchain` as `stdlib` version for `go.mod` files (aquasecurity#7163) * release: v0.55.0 [main] (aquasecurity#7271) * fix(license): stop spliting a long license text (aquasecurity#7336) Signed-off-by: knqyf263 <knqyf263@gmail.com> Co-authored-by: knqyf263 <knqyf263@gmail.com> * refactor(java): add error/statusCode for logs when we can't get pom.xml/maven-metadata.xml from remote repo (aquasecurity#7451) * chore(helm): bump up Trivy Helm chart (aquasecurity#7441) * chore(deps): bump the common group across 1 directory with 19 updates (aquasecurity#7436) Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: knqyf263 <knqyf263@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: knqyf263 <knqyf263@gmail.com> * chore(deps): bump the aws group with 6 updates (aquasecurity#7468) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix(oracle): Update EOL date for Oracle 7 (aquasecurity#7480) * fix(report): change a receiver of MarshalJSON (aquasecurity#7483) Signed-off-by: knqyf263 <knqyf263@gmail.com> * fix(report): fix error with unmarshal of `ExperimentalModifiedFindings` (aquasecurity#7463) Signed-off-by: knqyf263 <knqyf263@gmail.com> Co-authored-by: knqyf263 <knqyf263@gmail.com> * docs(oci): Add a note About the expected Media Type for the Trivy-DB OCI Artifact (aquasecurity#7449) * feat(license): improve license normalization (aquasecurity#7131) Signed-off-by: knqyf263 <knqyf263@gmail.com> Co-authored-by: DmitriyLewen <dmitriy.lewen@smartforce.io> Co-authored-by: knqyf263 <knqyf263@gmail.com> * docs(db): add a manifest example (aquasecurity#7485) Signed-off-by: knqyf263 <knqyf263@gmail.com> * revert(java): stop supporting of `test` scope for `pom.xml` files (aquasecurity#7488) * docs: refine go docs (aquasecurity#7442) Signed-off-by: knqyf263 <knqyf263@gmail.com> Co-authored-by: knqyf263 <knqyf263@gmail.com> * chore(vex): suppress openssl vulnerabilities (aquasecurity#7500) Signed-off-by: knqyf263 <knqyf263@gmail.com> * chore(deps): bump alpine from 3.20.0 to 3.20.3 (aquasecurity#7508) * chore(vex): add `CVE-2024-34155`, `CVE-2024-34156` and `CVE-2024-34158` in `trivy.openvex.json` (aquasecurity#7510) * fix(java): use `dependencyManagement` from root/child pom's for dependencies from parents (aquasecurity#7497) * refactor: split `.egg` and `packaging` analyzers (aquasecurity#7514) * feat(misconf): Register checks only when needed (aquasecurity#7435) * fix(misconf): Fix logging typo (aquasecurity#7473) * chore(deps): bump go-ebs-file (aquasecurity#7513) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * fix(sbom): parse type `framework` as `library` when unmarshalling `CycloneDX` files (aquasecurity#7527) * refactor(misconf): pass options to Rego scanner as is (aquasecurity#7529) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * fix(sbom): export bom-ref when converting a package to a component (aquasecurity#7340) Signed-off-by: knqyf263 <knqyf263@gmail.com> Co-authored-by: amf <amf@macbook.local> Co-authored-by: knqyf263 <knqyf263@gmail.com> * perf(misconf): use port ranges instead of enumeration (aquasecurity#7549) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * fix(misconf): Fixed scope for China Cloud (aquasecurity#7560) * docs(misconf): Add more info on how to use arbitrary JSON/YAML scan feat (aquasecurity#7458) * chore(deps): remove broken replaces for opa and discovery (aquasecurity#7600) * ci: cache test images for `integration`, `VM` and `module` tests (aquasecurity#7599) * ci: add `workflow_dispatch` trigger for test workflow. (aquasecurity#7606) * chore(deps): bump the common group across 1 directory with 20 updates (aquasecurity#7604) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: knqyf263 <knqyf263@gmail.com> * fix(db): check `DownloadedAt` for `trivy-java-db` (aquasecurity#7592) * fix: allow access to '..' in mapfs (aquasecurity#7575) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * test: use a local registry for remote scanning (aquasecurity#7607) Signed-off-by: knqyf263 <knqyf263@gmail.com> * fix(misconf): escape all special sequences (aquasecurity#7558) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * feat(misconf): add ability to disable checks by ID (aquasecurity#7536) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> Co-authored-by: Simar <simar@linux.com> * feat(suse): added SUSE Linux Enterprise Micro support (aquasecurity#7294) Signed-off-by: Marcus Meissner <meissner@suse.de> Signed-off-by: knqyf263 <knqyf263@gmail.com> Co-authored-by: knqyf263 <knqyf263@gmail.com> * fix(misconf): disable DS016 check for image history analyzer (aquasecurity#7540) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * ci: split `save` and `restore` cache actions (aquasecurity#7614) * refactor: fix auth error handling (aquasecurity#7615) Signed-off-by: knqyf263 <knqyf263@gmail.com> * feat(secret): enhance secret scanning for python binary files (aquasecurity#7223) Signed-off-by: knqyf263 <knqyf263@gmail.com> Co-authored-by: knqyf263 <knqyf263@gmail.com> * feat(java): add empty versions if `pom.xml` dependency versions can't be detected (aquasecurity#7520) Co-authored-by: Teppei Fukuda <knqyf263@gmail.com> * test: use loaded image names (aquasecurity#7617) Signed-off-by: knqyf263 <knqyf263@gmail.com> * ci: don't use cache for `setup-go` (aquasecurity#7622) * feat: support multiple DB repositories for vulnerability and Java DB (aquasecurity#7605) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * feat(misconf): Support `--skip-*` for all included modules (aquasecurity#7579) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> Co-authored-by: nikpivkin <nikita.pivkin@smartforce.io> * chore: add prefixes to log messages (aquasecurity#7625) Signed-off-by: knqyf263 <knqyf263@gmail.com> Co-authored-by: simar7 <1254783+simar7@users.noreply.github.com> * fix(misconf): Disable deprecated checks by default (aquasecurity#7632) * chore(deps): Bump trivy-checks to v1.1.0 (aquasecurity#7631) * fix(secret): change grafana token regex to find them without unquoted (aquasecurity#7627) * feat: support RPM archives (aquasecurity#7628) Signed-off-by: knqyf263 <knqyf263@gmail.com> * fix(misconf): not to warn about missing selectors of libraries (aquasecurity#7638) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> * release: v0.56.0 [main] (aquasecurity#7447) * fix(db): fix javadb downloading error handling [backport: release/v0.56] (aquasecurity#7646) Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> Co-authored-by: Nikita Pivkin <nikita.pivkin@smartforce.io> * release: v0.56.1 [release/v0.56] (aquasecurity#7648) * fix(sbom): add options for DBs in private registries [backport: release/v0.56] (aquasecurity#7691) Signed-off-by: knqyf263 <knqyf263@gmail.com> Co-authored-by: Teppei Fukuda <knqyf263@gmail.com> * fix(redhat): include arch in PURL qualifiers [backport: release/v0.56] (aquasecurity#7702) Signed-off-by: knqyf263 <knqyf263@gmail.com> Co-authored-by: Teppei Fukuda <knqyf263@gmail.com> * release: v0.56.2 [release/v0.56] (aquasecurity#7694) * Make liveness probe configurable (#3) --------- Signed-off-by: yusuke.koyoshi <yusuke.koyoshi@bizreach.co.jp> Signed-off-by: knqyf263 <knqyf263@gmail.com> Signed-off-by: nikpivkin <nikita.pivkin@smartforce.io> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Kevin Conner <kev.conner@getupcloud.com> Signed-off-by: Bob Callaway <bcallaway@google.com> Signed-off-by: Marcus Meissner <meissner@suse.de> Co-authored-by: yusuke-koyoshi <92022336+yusuke-koyoshi@users.noreply.github.com> Co-authored-by: Teppei Fukuda <knqyf263@gmail.com> Co-authored-by: Aruneko <yuki.fujita@bizreach.co.jp> Co-authored-by: Colm O hEigeartaigh <coheigea@users.noreply.github.com> Co-authored-by: DmitriyLewen <dmitriy.lewen@smartforce.io> Co-authored-by: afdesk <work@afdesk.com> Co-authored-by: Nikita Pivkin <nikita.pivkin@smartforce.io> Co-authored-by: Alberto Donato <albertodonato@users.noreply.github.com> Co-authored-by: simar7 <1254783+simar7@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Itay Shakury <itay@itaysk.com> Co-authored-by: DmitriyLewen <91113035+DmitriyLewen@users.noreply.github.com> Co-authored-by: aasish-r <aasishrampalli1997@gmail.com> Co-authored-by: Ori <59772293+orizerah@users.noreply.github.com> Co-authored-by: Kevin Conner <kev.conner@gmail.com> Co-authored-by: Bob Callaway <bobcallaway@users.noreply.github.com> Co-authored-by: vhash <29121316+LucasVanHaaren@users.noreply.github.com> Co-authored-by: psibre <psibre@users.noreply.github.com> Co-authored-by: Aqua Security automated builds <54269356+aqua-bot@users.noreply.github.com> Co-authored-by: s-reddy1498 <41355782+s-reddy1498@users.noreply.github.com> Co-authored-by: Squiddim <82903357+Squiddim@users.noreply.github.com> Co-authored-by: Pierre Baumard <pierre.baumard@cnav.fr> Co-authored-by: Lior Kaplan <lior@kaplanopensource.co.il> Co-authored-by: amf <amf@macbook.local> Co-authored-by: bloomadcariad <adam.bloom@cariad.us> Co-authored-by: Sylvain Baubeau <lebauce@gmail.com> Co-authored-by: Simar <simar@linux.com> Co-authored-by: Marcus Meissner <meissner@suse.de> Co-authored-by: Samuel Gaist <samuel.gaist@idiap.ch>

Signed-off-by: knqyf263 <knqyf263@gmail.com> Co-authored-by: DmitriyLewen <dmitriy.lewen@smartforce.io> Co-authored-by: knqyf263 <knqyf263@gmail.com>

(feat)Improve license normalization

44b0164

- Remove "THE " prefix - Remove "LICENSE " suffix - Fix lowercase key - Add BSD LICENSE 3 - Add ECLIPSE PUBLIC LICENSE 1.0 - Add MIT-0 - Add tests

pbaumard requested a review from knqyf263 as a code owner July 9, 2024 15:04

Fix indenting

5adc524

knqyf263 requested a review from DmitriyLewen July 9, 2024 17:41

DmitriyLewen reviewed Jul 10, 2024

View reviewed changes

Fix format and test

6b52af5

DmitriyLewen reviewed Jul 11, 2024

View reviewed changes

pkg/licensing/normalize_test.go Outdated Show resolved Hide resolved

mapping back to private and other suffixes ignore from SPDX

4dc5070

DmitriyLewen reviewed Jul 12, 2024

View reviewed changes

pkg/licensing/normalize.go Show resolved Hide resolved

pbaumard added 11 commits July 12, 2024 10:10

Do not change unmapped license name

60d29b0

Additional mappings from oss-review-toolkit and space normalization

ab62262

Fix double space

d0101fc

Add back GPL WITH AUTOCONF EXCEPTION

cd7d779

Normalize CycloneDX licenses and fix tests

aa7ee3c

Add MPL mappings

c21e723

Update golden test files

3055e10

Normalize suffixes using existing expression package

6c168f0

Fix copyright_test

6ec91a5

Add ValidateDocument in SPDX test

0f4c8cb

Fix linter

907683d

pbaumard requested a review from DmitriyLewen July 18, 2024 08:52

DmitriyLewen reviewed Jul 24, 2024

View reviewed changes

pbaumard added 5 commits July 24, 2024 15:21

Revert since only Name and Version are checked

5074f24

Revert cycloneDX normalization

8e6f9d3

Normalized SPDX license should be in Id field.

More tests

b8bb817

Comments

56093d9

Fix golden

85b8781

test(integration): update golden files

39f796e

DmitriyLewen reviewed Jul 25, 2024

View reviewed changes

Fix apk license parsing and use version regexp for license normalization

5f7cf6d

pbaumard requested a review from DmitriyLewen July 26, 2024 16:47

DmitriyLewen reviewed Jul 31, 2024

View reviewed changes

Check map in test; make map directly & move TestLaxSplitLicense to no…

aff6ebd

…rmalize_test

pbaumard requested a review from DmitriyLewen August 1, 2024 11:05

DmitriyLewen reviewed Aug 20, 2024

View reviewed changes

pbaumard and others added 3 commits September 2, 2024 17:01

Add field names in test

0134266

Merge branch 'aquasecurity:main' into feature/better-license-normalize

b85b781

Golden test update

8663dcd

pbaumard requested a review from DmitriyLewen September 3, 2024 09:10

Merge remote-tracking branch 'origin/main' into feature/better-licens…

932e937

…e-normalize

DmitriyLewen approved these changes Sep 11, 2024

View reviewed changes

test: add a file for private functions

f3f7c39

Signed-off-by: knqyf263 <knqyf263@gmail.com>

knqyf263 approved these changes Sep 11, 2024

View reviewed changes

knqyf263 enabled auto-merge September 11, 2024 06:30

knqyf263 added this pull request to the merge queue Sep 11, 2024

Merged via the queue into aquasecurity:main with commit 6472e3c Sep 11, 2024
12 checks passed

pbaumard deleted the feature/better-license-normalize branch September 11, 2024 13:06

aqua-bot mentioned this pull request Sep 11, 2024

release: v0.56.0 [main] #7447

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(license): improve license normalization #7131

feat(license): improve license normalization #7131

pbaumard commented Jul 9, 2024 •

edited

Loading

CLAassistant commented Jul 9, 2024 •

edited

Loading

CLAassistant commented Jul 9, 2024

DmitriyLewen left a comment

DmitriyLewen left a comment

DmitriyLewen Jul 24, 2024

pbaumard Jul 24, 2024

DmitriyLewen Jul 25, 2024

pbaumard Jul 26, 2024

DmitriyLewen Jul 24, 2024

pbaumard Jul 24, 2024

DmitriyLewen Jul 24, 2024

pbaumard Jul 24, 2024

DmitriyLewen Jul 25, 2024

DmitriyLewen Jul 25, 2024

DmitriyLewen Jul 25, 2024

DmitriyLewen Jul 25, 2024

pbaumard Jul 26, 2024

DmitriyLewen Jul 31, 2024

pbaumard Aug 1, 2024

DmitriyLewen Jul 31, 2024

DmitriyLewen Jul 31, 2024

pbaumard Aug 1, 2024

DmitriyLewen Jul 31, 2024

pbaumard Aug 1, 2024

DmitriyLewen Jul 31, 2024

pbaumard Aug 1, 2024

DmitriyLewen Jul 31, 2024

pbaumard Aug 1, 2024

DmitriyLewen left a comment

DmitriyLewen Aug 20, 2024

pbaumard Sep 2, 2024

DmitriyLewen left a comment

knqyf263 left a comment •

edited

Loading

	func Normalize(name string) string {
	return NormalizeLicense(name).String()
	}

	for i := 0; i < len(expectedPkgs); i++ {
	require.Equal(t, expectedPkgs[i].Name, detail.Packages[i].Name, tc.name)
	require.Equal(t, expectedPkgs[i].Version, detail.Packages[i].Version, tc.name)
	}

	assert.Empty(t, licensing.InvalidMappingKeys((nil)))
	assert.Empty(t, licensing.InvalidMappingKeys(nil))

feat(license): improve license normalization #7131

feat(license): improve license normalization #7131

Conversation

pbaumard commented Jul 9, 2024 • edited Loading

Description

Related issues

Checklist

CLAassistant commented Jul 9, 2024 • edited Loading

CLAassistant commented Jul 9, 2024

DmitriyLewen left a comment

Choose a reason for hiding this comment

DmitriyLewen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DmitriyLewen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DmitriyLewen left a comment

Choose a reason for hiding this comment

knqyf263 left a comment • edited Loading

Choose a reason for hiding this comment

pbaumard commented Jul 9, 2024 •

edited

Loading

CLAassistant commented Jul 9, 2024 •

edited

Loading

knqyf263 left a comment •

edited

Loading