Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add license checking tool #11398

Merged
merged 27 commits into from
Jul 28, 2022
Merged
Show file tree
Hide file tree
Changes from 26 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
d0cf3e9
Add license checker.
srebhan Jun 28, 2022
544e1cf
Add whitelist.
srebhan Jun 28, 2022
9092f70
Implement expiry of cache.
srebhan Jun 28, 2022
5fe137a
Add license checker build to makefile.
srebhan Jun 28, 2022
7f5a259
Fix license information for 'github.com/Azure/azure-sdk-for-go'
srebhan Jun 28, 2022
7beabea
Warn if whitelist entry does not match the expectations.
srebhan Jun 28, 2022
acb3a44
Whitelist 'github.com/ClickHouse/clickhouse-go' as license changed be…
srebhan Jun 28, 2022
b74e90d
Fix link for and whitelist 'github.com/couchbase/goutils' as license …
srebhan Jun 28, 2022
7862a30
Fix link for 'github.com/couchbase/goutils'.
srebhan Jun 28, 2022
47ef248
Fix license link for 'github.com/harlow/kinesis-consumer'.
srebhan Jun 28, 2022
26c0ecd
Fix license information for 'github.com/hashicorp/go-hclog'.
srebhan Jun 28, 2022
0379ee6
Fix license link for 'github.com/hashicorp/go-immutable-radix'.
srebhan Jun 28, 2022
dddef8b
Fix license link for 'github.com/josharian/intern'.
srebhan Jun 28, 2022
cea1501
Move license-name to SPDX mapping to file and add BSD-2-Clause-Views.
srebhan Jun 28, 2022
1a46b65
Fix license information for 'github.com/rcrowley/go-metrics'.
srebhan Jun 28, 2022
04b7a06
Fix license information for 'gopkg.in/yaml.v3'.
srebhan Jun 28, 2022
6b0f568
Fix license link for 'github.com/DataDog/datadog-agent'.
srebhan Jun 28, 2022
358e064
Sort whitelist.
srebhan Jun 28, 2022
0c74f85
Fix linter issues.
srebhan Jun 28, 2022
522e4bd
Sort packages in go.mod.
srebhan Jun 28, 2022
9b3f740
Add better help text and make the path optional.
srebhan Jun 29, 2022
2246391
Add README.md.
srebhan Jun 29, 2022
f95a382
Fix linter issues.
srebhan Jun 29, 2022
b44f8ca
Remove cache.
srebhan Jul 27, 2022
53c1519
Add SPDX mapping for EPL-2.0.
srebhan Jul 27, 2022
24f675d
Add license checking to nightly build.
srebhan Jul 27, 2022
30271d1
Remove unnecessary CircleCI persistence.
srebhan Jul 28, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -232,6 +232,25 @@ jobs:
os: windows
gotestsum: "gotestsum.exe"

test-licenses:
executor: telegraf-ci
steps:
- checkout
- restore_cache:
key: go-mod-v1-{{ checksum "go.sum" }}
- check-changed-files-or-halt
- run: 'make build_tools'
- run: 'sh ./tools/license_checker/license_checker -whitelist ./tools/license_checker/data/whitelist'
- save_cache:
name: 'go module cache'
key: go-mod-v1-{{ checksum "go.sum" }}
paths:
- '/go/pkg/mod'
- persist_to_workspace:
root: '/go'
paths:
- '*'
srebhan marked this conversation as resolved.
Show resolved Hide resolved

windows-package:
parameters:
nightly:
Expand Down Expand Up @@ -725,6 +744,7 @@ workflows:
- 'test-go-linux-386'
- 'test-go-mac'
- 'test-go-windows'
- 'test-licenses'
- 'windows-package':
name: 'windows-package-nightly'
nightly: true
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
/telegraf.exe
/telegraf.gz
/tools/package_lxd_test/package_lxd_test
/tools/license_checker/license_checker
/tools/readme_config_includer/generator
/vendor
.DS_Store
Expand Down
3 changes: 3 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,7 @@ versioninfo:
go generate cmd/telegraf/telegraf_windows.go; \

build_tools:
$(HOSTGO) build -o ./tools/license_checker/license_checker ./tools/license_checker
$(HOSTGO) build -o ./tools/readme_config_includer/generator ./tools/readme_config_includer/generator.go

embed_readme_%:
Expand Down Expand Up @@ -227,6 +228,8 @@ clean:
rm -rf tools/readme_config_includer/generator.exe
rm -rf tools/package_lxd_test/package_lxd_test
rm -rf tools/package_lxd_test/package_lxd_test.exe
rm -rf tools/license_checker/license_checker
rm -rf tools/license_checker/license_checker.exe

.PHONY: docker-image
docker-image:
Expand Down
16 changes: 8 additions & 8 deletions docs/LICENSE_OF_DEPENDENCIES.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ following works:
- github.com/Azure/azure-event-hubs-go [MIT License](https://github.com/Azure/azure-event-hubs-go/blob/master/LICENSE)
- github.com/Azure/azure-kusto-go [MIT License](https://github.com/Azure/azure-kusto-go/blob/master/LICENSE)
- github.com/Azure/azure-pipeline-go [MIT License](https://github.com/Azure/azure-pipeline-go/blob/master/LICENSE)
- github.com/Azure/azure-sdk-for-go [Apache License 2.0](https://github.com/Azure/azure-sdk-for-go/blob/master/LICENSE)
- github.com/Azure/azure-sdk-for-go [MIT License](https://github.com/Azure/azure-sdk-for-go/blob/main/LICENSE.txt)
- github.com/Azure/azure-sdk-for-go/sdk/azcore [MIT License](https://github.com/Azure/azure-sdk-for-go/blob/main/sdk/azcore/LICENSE.txt)
- github.com/Azure/azure-sdk-for-go/sdk/internal [MIT License](https://github.com/Azure/azure-sdk-for-go/blob/main/sdk/internal/LICENSE.txt)
- github.com/Azure/azure-sdk-for-go/sdk/storage/azblob [MIT License](https://github.com/Azure/azure-sdk-for-go/blob/main/sdk/storage/azblob/LICENSE.txt)
Expand Down Expand Up @@ -143,12 +143,12 @@ following works:
- github.com/grid-x/serial [MIT License](https://github.com/grid-x/serial/blob/master/LICENSE)
- github.com/gwos/tcg/sdk [MIT License](https://github.com/gwos/tcg/blob/master/LICENSE)
- github.com/hailocab/go-hostpool [MIT License](https://github.com/hailocab/go-hostpool/blob/master/LICENSE)
- github.com/harlow/kinesis-consumer [MIT License](https://github.com/harlow/kinesis-consumer/blob/master/MIT-LICENSE)
- github.com/harlow/kinesis-consumer [MIT License](https://github.com/harlow/kinesis-consumer/blob/master/LICENSE)
- github.com/hashicorp/consul/api [Mozilla Public License 2.0](https://github.com/hashicorp/consul/blob/master/LICENSE)
- github.com/hashicorp/errwrap [Mozilla Public License 2.0](https://github.com/hashicorp/errwrap/blob/master/LICENSE)
- github.com/hashicorp/go-cleanhttp [Mozilla Public License 2.0](https://github.com/hashicorp/go-cleanhttp/blob/master/LICENSE)
- github.com/hashicorp/go-hclog [Mozilla Public License 2.0](https://github.com/hashicorp/go-hclog/LICENSE)
- github.com/hashicorp/go-immutable-radix [Mozilla Public License 2.0](https://github.com/hashicorp/go-immutable-radix/LICENSE)
- github.com/hashicorp/go-hclog [MIT License](https://github.com/hashicorp/go-hclog/blob/main/LICENSE)
- github.com/hashicorp/go-immutable-radix [Mozilla Public License 2.0](https://github.com/hashicorp/go-immutable-radix/blob/master/LICENSE)
- github.com/hashicorp/go-multierror [Mozilla Public License 2.0](https://github.com/hashicorp/go-multierror/blob/master/LICENSE)
- github.com/hashicorp/go-rootcerts [Mozilla Public License 2.0](https://github.com/hashicorp/go-rootcerts/blob/master/LICENSE)
- github.com/hashicorp/go-uuid [Mozilla Public License 2.0](https://github.com/hashicorp/go-uuid/blob/master/LICENSE)
Expand Down Expand Up @@ -180,7 +180,7 @@ following works:
- github.com/jcmturner/rpc [Apache License 2.0](https://github.com/jcmturner/rpc/blob/master/LICENSE)
- github.com/jhump/protoreflect [Apache License 2.0](https://github.com/jhump/protoreflect/blob/master/LICENSE)
- github.com/jmespath/go-jmespath [Apache License 2.0](https://github.com/jmespath/go-jmespath/blob/master/LICENSE)
- github.com/josharian/intern [MIT License](https://github.com/josharian/intern/blob/master/license.md)
- github.com/josharian/intern [MIT License](https://github.com/josharian/intern/blob/master/LICENSE.md)
- github.com/josharian/native [MIT License](https://github.com/josharian/native/blob/main/license)
- github.com/jpillora/backoff [MIT License](https://github.com/jpillora/backoff/blob/master/LICENSE)
- github.com/json-iterator/go [MIT License](https://github.com/json-iterator/go/blob/master/LICENSE)
Expand Down Expand Up @@ -244,7 +244,7 @@ following works:
- github.com/prometheus/procfs [Apache License 2.0](https://github.com/prometheus/procfs/blob/master/LICENSE)
- github.com/prometheus/prometheus [Apache License 2.0](https://github.com/prometheus/prometheus/blob/master/LICENSE)
- github.com/rabbitmq/amqp091-go [BSD 2-Clause "Simplified" License](https://github.com/rabbitmq/amqp091-go/blob/main/LICENSE)
- github.com/rcrowley/go-metrics [MIT License](https://github.com/rcrowley/go-metrics/blob/master/LICENSE)
- github.com/rcrowley/go-metrics [BSD 2-Clause with views sentence](https://github.com/rcrowley/go-metrics/blob/master/LICENSE)
- github.com/remyoudompheng/bigfft [BSD 3-Clause "New" or "Revised" License](https://github.com/remyoudompheng/bigfft/blob/master/LICENSE)
- github.com/riemann/riemann-go-client [MIT License](https://github.com/riemann/riemann-go-client/blob/master/LICENSE)
- github.com/robbiet480/go.nut [MIT License](https://github.com/robbiet480/go.nut/blob/master/LICENSE)
Expand Down Expand Up @@ -316,7 +316,7 @@ following works:
- gopkg.in/tomb.v1 [BSD 3-Clause Clear License](https://github.com/go-tomb/tomb/blob/v1/LICENSE)
- gopkg.in/tomb.v2 [BSD 3-Clause Clear License](https://github.com/go-tomb/tomb/blob/v2/LICENSE)
- gopkg.in/yaml.v2 [Apache License 2.0](https://github.com/go-yaml/yaml/blob/v2.2.2/LICENSE)
- gopkg.in/yaml.v3 [Apache License 2.0](https://github.com/go-yaml/yaml/blob/v3/LICENSE)
- gopkg.in/yaml.v3 [MIT License](https://github.com/go-yaml/yaml/blob/v3/LICENSE)
- k8s.io/api [Apache License 2.0](https://github.com/kubernetes/client-go/blob/master/LICENSE)
- k8s.io/apimachinery [Apache License 2.0](https://github.com/kubernetes/client-go/blob/master/LICENSE)
- k8s.io/client-go [Apache License 2.0](https://github.com/kubernetes/client-go/blob/master/LICENSE)
Expand All @@ -333,4 +333,4 @@ following works:

## Telegraf used and modified code from these projects

- github.com/DataDog/datadog-agent [Apache License 2.0](https://github.com/DataDog/datadog-agent/LICENSE)
- github.com/DataDog/datadog-agent [Apache License 2.0](https://github.com/DataDog/datadog-agent/blob/main/LICENSE)
3 changes: 2 additions & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ require (
github.com/google/gnxi v0.0.0-20220411075422-cd6b043b7fd0
github.com/google/go-cmp v0.5.8
github.com/google/go-github/v32 v32.1.0
github.com/google/licensecheck v0.3.1
github.com/google/uuid v1.3.0
github.com/gopcua/opcua v0.3.3
github.com/gophercloud/gophercloud v0.25.0
Expand Down Expand Up @@ -155,6 +156,7 @@ require (
go.opentelemetry.io/otel/metric v0.30.0
go.opentelemetry.io/otel/sdk/metric v0.28.0
go.starlark.net v0.0.0-20220328144851-d1966c6b9fcd
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4
golang.org/x/net v0.0.0-20220622184535-263ec571b305
golang.org/x/oauth2 v0.0.0-20220608161450-d0670ef3b1eb
golang.org/x/sync v0.0.0-20220601150217-0de741cfad7f
Expand Down Expand Up @@ -375,7 +377,6 @@ require (
go.uber.org/multierr v1.6.0 // indirect
golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d // indirect
golang.org/x/exp v0.0.0-20200513190911-00229845015e // indirect
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4 // indirect
golang.org/x/term v0.0.0-20220526004731-065cf7ba2467 // indirect
golang.org/x/time v0.0.0-20220609170525-579cf78fd858 // indirect
golang.org/x/tools v0.1.11 // indirect
Expand Down
2 changes: 2 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -1105,6 +1105,8 @@ github.com/google/go-querystring v1.0.0/go.mod h1:odCYkC5MyYFN7vkCjXpyrEuKhc/BUO
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/gofuzz v1.1.0 h1:Hsa8mG0dQ46ij8Sl2AYJDUv1oA9/d6Vk+3LG99Oe02g=
github.com/google/gofuzz v1.1.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/licensecheck v0.3.1 h1:QoxgoDkaeC4nFrtGN1jV7IPmDCHFNIVh54e5hSt6sPs=
github.com/google/licensecheck v0.3.1/go.mod h1:ORkR35t/JjW+emNKtfJDII0zlciG9JgbT7SmsohlHmY=
github.com/google/martian v2.1.0+incompatible h1:/CP5g8u/VJHijgedC/Legn3BAbAaWPgecwXBIDzw5no=
github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
github.com/google/martian/v3 v3.0.0/go.mod h1:y5Zk1BBys9G+gd6Jrk0W3cC1+ELVxBWuIGO+w/tUAp0=
Expand Down
93 changes: 93 additions & 0 deletions tools/license_checker/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
# Dependency license verification tool

This tool allows the verification of information in
`docs/LICENSE_OF_DEPENDENCIES.md` against the linked license
information. To do so, the license reported by the user is
checked against the license classification of the downloaded
license file for each dependency.

## Building

```shell
make build_tools
```

## Running

The simplest way to run the verification tool is to execute

```shell
telegraf$ ./tools/license_checker/license_checker
```

using the current directory as telegraf's root directory and verifies
all licenses. Only errors will be reported by default.

There are multiple options you can use to customize the verification.
Take a look at

```shell
telegraf$ ./tools/license_checker/license_checker --help
```

to get an overview.

As the verification tool downloads each license file linked in the
dependency license document, you should be careful on not exceeding
the access limits of e.g. GitHub by running the tool too frequent.

Some packages change the license for newer versions. As we always
link to the latest license text the classification might not match
the actual license of our used dependency. Furthermore, some license
text might be wrongly classified, or not classified at all. In these
cases, you can use a _whitelist_ to explicitly state the license
SPDX classifier for those packages.
See the [whitelist section](#whitelist) for more details.

The recommended use in telegraf is to run

```shell
telegraf$ ./tools/license_checker/license_checker \
-whitelist ./tools/license_checker/data/whitelist
```

using the code-versioned whitelist. This command will report all
non-matching entries with an `ERR:` prefix.

## Whitelist

Whitelist entries contain explicit license information for
a set of packages to use instead of classification. Each entry
in the whitelist is a line of the form

```text
[comparison operator]<package name>[@vX.Y.Z] <license SPDX>
```

where the _comparison operator_ is one of `>`, `>=`, `=`, `<=` or `<`
and the _license SPDX_ is a [SPDX license identifier][spdx].
In case no package version is specified, the entry matches all versions
of the library. Furthermore, the comparison operator can be omitted
which is equivalent to an exact match (`=`).

The entries are processed in order until the first match is found.

Here is an example of a whitelist. Assume that you have library
`github.com/foo/bar` which started out with the `MIT` license
until version 1.0.0 where it changed to `EFL-1.0` until it again
changed to `EFL-2.0` starting __after__ version 2.3.0. In this case
the whitelist should look like this

```text
<github.com/foo/bar@v1.0.0 MIT
<=github.com/foo/bar@v2.3.0 EFL-1.0
github.com/foo/bar EFL-2.0
```

All versions below 1.0.0 are matched by the first line and are thus
classified as `MIT`. The second line matches everything that is
above 1.0.0 (thus not matched by the first line) until (and including)
2.3.0. The last line with catch everything that was passing the first
two lines i.e. everything after 2.3.0.

[spdx]: https://spdx.org/licenses/
15 changes: 15 additions & 0 deletions tools/license_checker/data/spdx_mapping.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
{
"Apache License 2.0": "Apache-2.0",
"BSD 2-Clause with views sentence": "BSD-2-Clause-Views",
"BSD 2-Clause \"Simplified\" License": "BSD-2-Clause",
"BSD 3-Clause \"New\" or \"Revised\" License": "BSD-3-Clause",
"BSD 3-Clause Clear License": "BSD-3-Clause",
"BSD 3-Clause License": "BSD-3-Clause",
"Eclipse Public License - v 1.0": "EPL-1.0",
"Eclipse Public License - v 2.0": "EPL-2.0",
"ISC License": "ISC",
"MIT License": "MIT",
"Mozilla Public License 2.0": "MPL-2.0",
"The Unlicense": "Unlicense",
"zlib License": "Zlib"
}
3 changes: 3 additions & 0 deletions tools/license_checker/data/whitelist
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
<github.com/ClickHouse/clickhouse-go@v2.0.0 MIT
<github.com/couchbase/goutils@v0.1.2 Apache-2.0
<github.com/eclipse/paho.mqtt.golang@v1.4.0 EPL-1.0
Loading