Skip to content

Commit

Permalink
feat: Add license checking tool (#11398)
Browse files Browse the repository at this point in the history
  • Loading branch information
srebhan authored Jul 28, 2022
1 parent ff17ede commit fd84042
Show file tree
Hide file tree
Showing 12 changed files with 605 additions and 9 deletions.
16 changes: 16 additions & 0 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -232,6 +232,21 @@ jobs:
os: windows
gotestsum: "gotestsum.exe"

test-licenses:
executor: telegraf-ci
steps:
- checkout
- restore_cache:
key: go-mod-v1-{{ checksum "go.sum" }}
- check-changed-files-or-halt
- run: 'make build_tools'
- run: 'sh ./tools/license_checker/license_checker -whitelist ./tools/license_checker/data/whitelist'
- save_cache:
name: 'go module cache'
key: go-mod-v1-{{ checksum "go.sum" }}
paths:
- '/go/pkg/mod'

windows-package:
parameters:
nightly:
Expand Down Expand Up @@ -725,6 +740,7 @@ workflows:
- 'test-go-linux-386'
- 'test-go-mac'
- 'test-go-windows'
- 'test-licenses'
- 'windows-package':
name: 'windows-package-nightly'
nightly: true
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
/telegraf.exe
/telegraf.gz
/tools/package_lxd_test/package_lxd_test
/tools/license_checker/license_checker
/tools/readme_config_includer/generator
/vendor
.DS_Store
Expand Down
3 changes: 3 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,7 @@ versioninfo:
go generate cmd/telegraf/telegraf_windows.go; \

build_tools:
$(HOSTGO) build -o ./tools/license_checker/license_checker ./tools/license_checker
$(HOSTGO) build -o ./tools/readme_config_includer/generator ./tools/readme_config_includer/generator.go

embed_readme_%:
Expand Down Expand Up @@ -227,6 +228,8 @@ clean:
rm -rf tools/readme_config_includer/generator.exe
rm -rf tools/package_lxd_test/package_lxd_test
rm -rf tools/package_lxd_test/package_lxd_test.exe
rm -rf tools/license_checker/license_checker
rm -rf tools/license_checker/license_checker.exe

.PHONY: docker-image
docker-image:
Expand Down
16 changes: 8 additions & 8 deletions docs/LICENSE_OF_DEPENDENCIES.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ following works:
- github.com/Azure/azure-event-hubs-go [MIT License](https://github.com/Azure/azure-event-hubs-go/blob/master/LICENSE)
- github.com/Azure/azure-kusto-go [MIT License](https://github.com/Azure/azure-kusto-go/blob/master/LICENSE)
- github.com/Azure/azure-pipeline-go [MIT License](https://github.com/Azure/azure-pipeline-go/blob/master/LICENSE)
- github.com/Azure/azure-sdk-for-go [Apache License 2.0](https://github.com/Azure/azure-sdk-for-go/blob/master/LICENSE)
- github.com/Azure/azure-sdk-for-go [MIT License](https://github.com/Azure/azure-sdk-for-go/blob/main/LICENSE.txt)
- github.com/Azure/azure-sdk-for-go/sdk/azcore [MIT License](https://github.com/Azure/azure-sdk-for-go/blob/main/sdk/azcore/LICENSE.txt)
- github.com/Azure/azure-sdk-for-go/sdk/internal [MIT License](https://github.com/Azure/azure-sdk-for-go/blob/main/sdk/internal/LICENSE.txt)
- github.com/Azure/azure-sdk-for-go/sdk/storage/azblob [MIT License](https://github.com/Azure/azure-sdk-for-go/blob/main/sdk/storage/azblob/LICENSE.txt)
Expand Down Expand Up @@ -143,12 +143,12 @@ following works:
- github.com/grid-x/serial [MIT License](https://github.com/grid-x/serial/blob/master/LICENSE)
- github.com/gwos/tcg/sdk [MIT License](https://github.com/gwos/tcg/blob/master/LICENSE)
- github.com/hailocab/go-hostpool [MIT License](https://github.com/hailocab/go-hostpool/blob/master/LICENSE)
- github.com/harlow/kinesis-consumer [MIT License](https://github.com/harlow/kinesis-consumer/blob/master/MIT-LICENSE)
- github.com/harlow/kinesis-consumer [MIT License](https://github.com/harlow/kinesis-consumer/blob/master/LICENSE)
- github.com/hashicorp/consul/api [Mozilla Public License 2.0](https://github.com/hashicorp/consul/blob/master/LICENSE)
- github.com/hashicorp/errwrap [Mozilla Public License 2.0](https://github.com/hashicorp/errwrap/blob/master/LICENSE)
- github.com/hashicorp/go-cleanhttp [Mozilla Public License 2.0](https://github.com/hashicorp/go-cleanhttp/blob/master/LICENSE)
- github.com/hashicorp/go-hclog [Mozilla Public License 2.0](https://github.com/hashicorp/go-hclog/LICENSE)
- github.com/hashicorp/go-immutable-radix [Mozilla Public License 2.0](https://github.com/hashicorp/go-immutable-radix/LICENSE)
- github.com/hashicorp/go-hclog [MIT License](https://github.com/hashicorp/go-hclog/blob/main/LICENSE)
- github.com/hashicorp/go-immutable-radix [Mozilla Public License 2.0](https://github.com/hashicorp/go-immutable-radix/blob/master/LICENSE)
- github.com/hashicorp/go-multierror [Mozilla Public License 2.0](https://github.com/hashicorp/go-multierror/blob/master/LICENSE)
- github.com/hashicorp/go-rootcerts [Mozilla Public License 2.0](https://github.com/hashicorp/go-rootcerts/blob/master/LICENSE)
- github.com/hashicorp/go-uuid [Mozilla Public License 2.0](https://github.com/hashicorp/go-uuid/blob/master/LICENSE)
Expand Down Expand Up @@ -180,7 +180,7 @@ following works:
- github.com/jcmturner/rpc [Apache License 2.0](https://github.com/jcmturner/rpc/blob/master/LICENSE)
- github.com/jhump/protoreflect [Apache License 2.0](https://github.com/jhump/protoreflect/blob/master/LICENSE)
- github.com/jmespath/go-jmespath [Apache License 2.0](https://github.com/jmespath/go-jmespath/blob/master/LICENSE)
- github.com/josharian/intern [MIT License](https://github.com/josharian/intern/blob/master/license.md)
- github.com/josharian/intern [MIT License](https://github.com/josharian/intern/blob/master/LICENSE.md)
- github.com/josharian/native [MIT License](https://github.com/josharian/native/blob/main/license)
- github.com/jpillora/backoff [MIT License](https://github.com/jpillora/backoff/blob/master/LICENSE)
- github.com/json-iterator/go [MIT License](https://github.com/json-iterator/go/blob/master/LICENSE)
Expand Down Expand Up @@ -244,7 +244,7 @@ following works:
- github.com/prometheus/procfs [Apache License 2.0](https://github.com/prometheus/procfs/blob/master/LICENSE)
- github.com/prometheus/prometheus [Apache License 2.0](https://github.com/prometheus/prometheus/blob/master/LICENSE)
- github.com/rabbitmq/amqp091-go [BSD 2-Clause "Simplified" License](https://github.com/rabbitmq/amqp091-go/blob/main/LICENSE)
- github.com/rcrowley/go-metrics [MIT License](https://github.com/rcrowley/go-metrics/blob/master/LICENSE)
- github.com/rcrowley/go-metrics [BSD 2-Clause with views sentence](https://github.com/rcrowley/go-metrics/blob/master/LICENSE)
- github.com/remyoudompheng/bigfft [BSD 3-Clause "New" or "Revised" License](https://github.com/remyoudompheng/bigfft/blob/master/LICENSE)
- github.com/riemann/riemann-go-client [MIT License](https://github.com/riemann/riemann-go-client/blob/master/LICENSE)
- github.com/robbiet480/go.nut [MIT License](https://github.com/robbiet480/go.nut/blob/master/LICENSE)
Expand Down Expand Up @@ -316,7 +316,7 @@ following works:
- gopkg.in/tomb.v1 [BSD 3-Clause Clear License](https://github.com/go-tomb/tomb/blob/v1/LICENSE)
- gopkg.in/tomb.v2 [BSD 3-Clause Clear License](https://github.com/go-tomb/tomb/blob/v2/LICENSE)
- gopkg.in/yaml.v2 [Apache License 2.0](https://github.com/go-yaml/yaml/blob/v2.2.2/LICENSE)
- gopkg.in/yaml.v3 [Apache License 2.0](https://github.com/go-yaml/yaml/blob/v3/LICENSE)
- gopkg.in/yaml.v3 [MIT License](https://github.com/go-yaml/yaml/blob/v3/LICENSE)
- k8s.io/api [Apache License 2.0](https://github.com/kubernetes/client-go/blob/master/LICENSE)
- k8s.io/apimachinery [Apache License 2.0](https://github.com/kubernetes/client-go/blob/master/LICENSE)
- k8s.io/client-go [Apache License 2.0](https://github.com/kubernetes/client-go/blob/master/LICENSE)
Expand All @@ -333,4 +333,4 @@ following works:

## Telegraf used and modified code from these projects

- github.com/DataDog/datadog-agent [Apache License 2.0](https://github.com/DataDog/datadog-agent/LICENSE)
- github.com/DataDog/datadog-agent [Apache License 2.0](https://github.com/DataDog/datadog-agent/blob/main/LICENSE)
3 changes: 2 additions & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ require (
github.com/google/gnxi v0.0.0-20220411075422-cd6b043b7fd0
github.com/google/go-cmp v0.5.8
github.com/google/go-github/v32 v32.1.0
github.com/google/licensecheck v0.3.1
github.com/google/uuid v1.3.0
github.com/gopcua/opcua v0.3.3
github.com/gophercloud/gophercloud v0.25.0
Expand Down Expand Up @@ -155,6 +156,7 @@ require (
go.opentelemetry.io/otel/metric v0.30.0
go.opentelemetry.io/otel/sdk/metric v0.28.0
go.starlark.net v0.0.0-20220328144851-d1966c6b9fcd
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4
golang.org/x/net v0.0.0-20220622184535-263ec571b305
golang.org/x/oauth2 v0.0.0-20220608161450-d0670ef3b1eb
golang.org/x/sync v0.0.0-20220601150217-0de741cfad7f
Expand Down Expand Up @@ -375,7 +377,6 @@ require (
go.uber.org/multierr v1.6.0 // indirect
golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d // indirect
golang.org/x/exp v0.0.0-20200513190911-00229845015e // indirect
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4 // indirect
golang.org/x/term v0.0.0-20220526004731-065cf7ba2467 // indirect
golang.org/x/time v0.0.0-20220609170525-579cf78fd858 // indirect
golang.org/x/tools v0.1.11 // indirect
Expand Down
2 changes: 2 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -1105,6 +1105,8 @@ github.com/google/go-querystring v1.0.0/go.mod h1:odCYkC5MyYFN7vkCjXpyrEuKhc/BUO
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/gofuzz v1.1.0 h1:Hsa8mG0dQ46ij8Sl2AYJDUv1oA9/d6Vk+3LG99Oe02g=
github.com/google/gofuzz v1.1.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/licensecheck v0.3.1 h1:QoxgoDkaeC4nFrtGN1jV7IPmDCHFNIVh54e5hSt6sPs=
github.com/google/licensecheck v0.3.1/go.mod h1:ORkR35t/JjW+emNKtfJDII0zlciG9JgbT7SmsohlHmY=
github.com/google/martian v2.1.0+incompatible h1:/CP5g8u/VJHijgedC/Legn3BAbAaWPgecwXBIDzw5no=
github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
github.com/google/martian/v3 v3.0.0/go.mod h1:y5Zk1BBys9G+gd6Jrk0W3cC1+ELVxBWuIGO+w/tUAp0=
Expand Down
93 changes: 93 additions & 0 deletions tools/license_checker/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
# Dependency license verification tool

This tool allows the verification of information in
`docs/LICENSE_OF_DEPENDENCIES.md` against the linked license
information. To do so, the license reported by the user is
checked against the license classification of the downloaded
license file for each dependency.

## Building

```shell
make build_tools
```

## Running

The simplest way to run the verification tool is to execute

```shell
telegraf$ ./tools/license_checker/license_checker
```

using the current directory as telegraf's root directory and verifies
all licenses. Only errors will be reported by default.

There are multiple options you can use to customize the verification.
Take a look at

```shell
telegraf$ ./tools/license_checker/license_checker --help
```

to get an overview.

As the verification tool downloads each license file linked in the
dependency license document, you should be careful on not exceeding
the access limits of e.g. GitHub by running the tool too frequent.

Some packages change the license for newer versions. As we always
link to the latest license text the classification might not match
the actual license of our used dependency. Furthermore, some license
text might be wrongly classified, or not classified at all. In these
cases, you can use a _whitelist_ to explicitly state the license
SPDX classifier for those packages.
See the [whitelist section](#whitelist) for more details.

The recommended use in telegraf is to run

```shell
telegraf$ ./tools/license_checker/license_checker \
-whitelist ./tools/license_checker/data/whitelist
```

using the code-versioned whitelist. This command will report all
non-matching entries with an `ERR:` prefix.

## Whitelist

Whitelist entries contain explicit license information for
a set of packages to use instead of classification. Each entry
in the whitelist is a line of the form

```text
[comparison operator]<package name>[@vX.Y.Z] <license SPDX>
```

where the _comparison operator_ is one of `>`, `>=`, `=`, `<=` or `<`
and the _license SPDX_ is a [SPDX license identifier][spdx].
In case no package version is specified, the entry matches all versions
of the library. Furthermore, the comparison operator can be omitted
which is equivalent to an exact match (`=`).

The entries are processed in order until the first match is found.

Here is an example of a whitelist. Assume that you have library
`github.com/foo/bar` which started out with the `MIT` license
until version 1.0.0 where it changed to `EFL-1.0` until it again
changed to `EFL-2.0` starting __after__ version 2.3.0. In this case
the whitelist should look like this

```text
<github.com/foo/bar@v1.0.0 MIT
<=github.com/foo/bar@v2.3.0 EFL-1.0
github.com/foo/bar EFL-2.0
```

All versions below 1.0.0 are matched by the first line and are thus
classified as `MIT`. The second line matches everything that is
above 1.0.0 (thus not matched by the first line) until (and including)
2.3.0. The last line with catch everything that was passing the first
two lines i.e. everything after 2.3.0.

[spdx]: https://spdx.org/licenses/
15 changes: 15 additions & 0 deletions tools/license_checker/data/spdx_mapping.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
{
"Apache License 2.0": "Apache-2.0",
"BSD 2-Clause with views sentence": "BSD-2-Clause-Views",
"BSD 2-Clause \"Simplified\" License": "BSD-2-Clause",
"BSD 3-Clause \"New\" or \"Revised\" License": "BSD-3-Clause",
"BSD 3-Clause Clear License": "BSD-3-Clause",
"BSD 3-Clause License": "BSD-3-Clause",
"Eclipse Public License - v 1.0": "EPL-1.0",
"Eclipse Public License - v 2.0": "EPL-2.0",
"ISC License": "ISC",
"MIT License": "MIT",
"Mozilla Public License 2.0": "MPL-2.0",
"The Unlicense": "Unlicense",
"zlib License": "Zlib"
}
3 changes: 3 additions & 0 deletions tools/license_checker/data/whitelist
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
<github.com/ClickHouse/clickhouse-go@v2.0.0 MIT
<github.com/couchbase/goutils@v0.1.2 Apache-2.0
<github.com/eclipse/paho.mqtt.golang@v1.4.0 EPL-1.0
Loading

0 comments on commit fd84042

Please sign in to comment.