Way to avoid pulling every dependency

[kevinburke1]

I am using archiver to pack and unpack .tar.gz files. For a project I am working on I am vendoring every library with go get -mod=vendor ./.... Unfortunately with archiver this means that I also must vendor the Brotli, Snappy etc. upstream libraries, even though the binary I am using doesn't actually link them or need them. This means an unnecessarily large vendor directory, larger Git pushes/pulls, etc.

Is there a way to only get the Targz struct out of this package? That way I wouldn't need to depend on/download the Brotli library.

[mholt]

How large is your vendor folder? My recommendation would simply be to fork the project and remove all the files for the formats you don't need.

[kevinburke1]

It's over 10MB at this point... one approach would be to mirror e.g. crypto/sha{1,256,512} where multiple different packages implement a single interface, and callers can either use the broad interface or the single package that they need.

…

On Sun, Sep 01, 2019 at 8:14 PM, Matt Holt < ***@***.*** > wrote: How large is your vendor folder? My recommendation would simply be to fork the project and remove all the files for the formats you don't need. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub ( #180?email_source=notifications&email_token=AML53FIWUGMSTYSHNCLR6XTQHSAIRA5CNFSM4ISYGJG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5US4WY#issuecomment-526986843 ) , or mute the thread ( https://github.com/notifications/unsubscribe-auth/AML53FL7BG7KFQADW7FSPELQHSAIRANCNFSM4ISYGJGQ ).

[mholt]

Okay, you're referring to a plugin model kind of like what Caddy does.

I don't have the time to work on that for this repository right now, but if it was a PR I could review it. (I am not sure how quickly I could get around to it, though.) The quickest short-term solution for you would probably be to do what I suggested and fork it, remove the features you don't need, and just use that for now.

[kevinburke1]

I'm imagining something where each different format (brotli, zip etc) would have its own package, and then the top-level Archiver could either import them via aliases or call Register() or something to get all of them. Most people could import Archiver, if you cared you could import the package you want directly and avoid the other dependencies.

I took a short stab at this and two problems come up right away:

• everything depends on *archiver.File and *archiver.FileInfo so if the subpackages have to import those from the main package, you're toast since then you get everything anyway. We could also move those to a subpackage - say, common or something - to avoid a circular import.
• same with the helpers like fileExists, which would also need to move to a subpackage.

[kevinburke1]

I took a stab at laying out what this could look like, let me know what you think.

https://github.com/mholt/archiver/compare/master...kevinburkemeter:package-layout?expand=1

[mholt]

Ah, yeah, so you want something like what Caddy does. Totally doable for next major version, but there is a high cost: it's a lot of work to build the application, since you essentially have to customize it for your own requirements. Then you have to answer questions like, what would the default/standard distribution look like: which packages are included, etc?

We have some tooling and infrastructure to help make this easier in Caddy, but it's still a pain (but given how many possible features people demand from their web servers, makes a lot of sense in Caddy's case).

I hate dependencies as much as the next person, but... is it worth it here?

Is the primary motivation for all this only to slim down the vendor folder?

[kevinburke1]

Basically, yeah, just reduce the number of requirements in the package. My sense was that the standard distribution would be everything that's currently imported by archiver - the archiver package would stay the same, it would just be possible, if you so desired, to only import the packages you wanted to.

[kevinburke1]

Here's what the commit pointing at the fork looks like. Most of the gains are from avoiding brotli since that's about 250,000 lines.

$ git amend
[zip-dep 3c4db0f] go.mod: use archiver fork
 Date: Mon Aug 10 12:31:32 2020 -0700
 187 files changed, 273 insertions(+), 283382 deletions(-)

[kevinburke1]

After this change our binary size dropped from 52MB to 48MB.

[mholt]

#302 effectively uses this "plugin" (or "format") registration system, which will become v4 of this package. While the Go module still has the dependencies of the various formats, we can in theory move them into separate packages or modules now if we wanted to. I just don't know if I'm motivated to go to all the work to separate them all out, and then have users import each one they want individually...

(Will close, but not because I'm rejecting it. If others adequately have this need too, we can reopen and discuss.)