native go modules support proposal (originally named v2 proposal)

[Bobgy]

Updates

2022-03-12: #70 (comment) This will be released as v1.1 instead, because the breaking change might not affect anyone now (TL;DR there's only a regression for people still using GOPATH). Also renamed this issue to make the purpose clearer.

2021-1-11: #70 (comment) I sent out a bunch of PRs for merging changes back to go-licenses repo. The final PRs I'm sending now try to change as minimal as possible. There are no substantial command/arg changes and there are no significant behavior changes. The only major theme is using go modules information to get license URLs more stably than the v1 go-licenses tool.

Original Proposal

Hi everyone, I'm building a rewrite of go-licenses that I propose to be go-licenses/v2.
Latest version on my dev branch: https://github.com/Bobgy/go-licenses/blob/main/v2. No longer latest.
Built binary releases: https://github.com/Bobgy/go-licenses/releases.

I initially built it as a new tool, but after chatting with @wlynch, we agreed that it would better be v2 of this tool rather than a new brand.

The major changes are:

• only supports go modules, it can get license URL for modules without a need for you to vendor your dependencies.
• do not assume each package has a single license, v2 will scan all the files for each module to find licenses. Deferred for future considerations.

For more details, refer to the README.

The new tool already addresses many open issues:

• traverse vanity urls for repository location for license url #32 vanity urls are curled and extracted to find the repo
• Repository url paths should be versioned #33 urls are now versioned according to go modules info
• Libraries are ignored when goroot and gopath share the same prefix #38 now we use modules info, no custom parsing
• Go modules support? #53 proper go modules support
• bug: Odd lack of dependency licenses for moby/buildkit #64 is a subproblem of proper go modules support

Deprioritized features

• go-licenses doesn't find license in UNLICENSE files in repos #39 we scan all files for licenses, so we can find UNLICENSE files too

and might help the following (but needs confirmation):

• golang.org/x/sys and other packages don't seem to work #18

Roadmap

• chore: add Makefile and a precommit check #101
• CSV command
  • Get modules/packages list
    • P0 v2: list dependencies from either import path or go binary #82
  • feat!(v2): use go modules info #94
  • feat!(v2): csv - validate license URL and guess more if invalid #110
  • Config file customization & overrides
• Documentation
  • doc: update usages for v1.1 #111
• Smaller Issues
  • P1 Replace glog comment

[Bobgy]

Process to v2

Initial PR: #67.
However, based on #67 (review).

Unfortunately this is a very large change. I've left some initial comments as a first pass, but we'll likely need to do multiple iterations to resolve everything. Something that would greatly help me out for reviewing is if we could break this up into separate PRs. Here's some individual chunks I see that might be good places to break this apart:

I'll be sending several PRs to gradually upstream the new tool instead.
While addressing comments, I'll be reducing the difference with go-licenses v1 and try to make upgrade easier.

Here, I'm using this issue to track progress and welcome any feedback & code reviews.

[Bobgy]

EDIT: Roadmap section moved to top comment.

[mythi]

Suggest: Save command: compress sources

[Bobgy]

I sent out a bunch of PRs which are minimal and focused on one thing, and a few more are ready (but I want to wait for the first PRs getting reviewed first).
The rough idea is starting from a copy of v1 go-licenses tool and gradually replace each command with the new implementation, so that we keep the interface and behavior backward compatible as much as possible.

[Bobgy]

Reply to #93 (review)

@wlynch let's track general discussion in the main issue?

General question before I start reviewing these PRs in more details - are the changes to csv/check changes introducing any changes w.r.t input/outputs of the commands, or is this strictly an upgrade using module info?

No, there are no substantial changes to io.

There is a minor and optional output format change of adding a space after each comma to make it convenient to click the license URL in terminal/editor, do you have any concerns?

Also, git-url argument is removed, because it's no longer used. (But I am reconsidering this, because it may be useful for the main module.)

If there isn't anything that would break compatibility with existing clients, it might make sense to just merge these into v1. 👀
The bits I was primarily concerned with the original v2 proposal were the new commands and the different output format.

Well, the major breaking change is that we no longer support non go module projects. That's probably worth a v2. What do you think?

[Bobgy]

Agreement today:

• let's use major branch approach of reaching v2, because the current v2 is a natural continuation of v1 with some minor breaking changes. Existing PRs may still be easy to merge into v2.

AI:

• @wlynch will tag a v1.0.0 release, so that people won't get a WIP v2 version accidentally in master branch
• @Bobgy will move PRs to target master branch.

We will re-discuss at what point we want to bump to v2 as the development continues.

[Bobgy]

@wlynch #95, #101 and #94 are ready for review : )

[Bobgy]

Note, I created a github release in my fork in case anyone want to try the proposed new tool before above PRs are merged: https://github.com/Bobgy/go-licenses/releases/tag/v0.1.0-2022-01-20

EDIT: my compiled binary did not quite work.

[Bobgy]

Created a new release that can be installed from source: https://github.com/Bobgy/go-licenses/releases/tag/v2.0.0-alpha.0. It's mostly ready to use and near backwards compatible.

To try my fork:

go get github.com/Bobgy/go-licenses/v2
go-licenses --help

[oxisto]

Is it somehow planned / possible to configure the check command, e.g. not only check for forbidden license but also for restricted? Or to configure the list of forbidden libraries? Or to at least print out the license type in the csv so I can grep for it? :)

[Bobgy]

Hi @oxisto, it's not in the scope. You might want to file a separate issue for this feature request.

[oxisto]

Hi @oxisto, it's not in the scope. You might want to file a separate issue for this feature request.

got it. thanks for your answer!

[Bobgy]

Update, based on discussion with @wlynch, we think that the breaking change for GOPATH might not be noticeable, because we don't know anyone still using GOPATH now. Therefore, we will consider releasing the next version as v1.1 instead. There won't be an actual v2.

Let us know if you use go-licenses and still use GOPATH, we can consider making changes in this proposal an actual v2 when needed.

[Bobgy]

The major changes have been released as v1.1.
Future improvements will be tracked individually in issues.