-
Notifications
You must be signed in to change notification settings - Fork 17.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
x/build/cmd/relui: reduce Go download sizes #27151
Comments
Untarring the release and gzip-compressing each file before doing du stats gives a reasonable way to measure the cost of removing bits:
Here are the top directories:
Biggest binaries:
We have 6.6 MB of testdata:
|
We ship three copies of Mark Twain's Tom Sawyer?
|
About the pkg files that get rebuilt, a common issue in the past has been users not having write permissions on the system GOROOT, hence breaking or rebuilding every time. Will that be a problem? |
@FiloSottile, that hasn't been a problem since at least Go 1.10. It writes to the build cache instead. You can experiment with the Dockerfile linked above. |
Currently we ship the tour so one can do This is all quite convoluted and I've seen reports of people confused by this. If we remove the tour from the binary releases we will also be able to simplify our instructions for running it locally: it's always "go get it, and then run the |
In that case I would like to remove the tour, maybe even for Go 1.11. |
/cc @katiehockman |
Change https://golang.org/cl/131156 mentions this issue: |
This removes the tour from the release binary for go1.12.x and later releases, lowering the size of each release. It will now redirect to tour.golang.org, unless the user runs "go get -u golang.org/x/tour" to cache it locally, in which case the local copy will be preferred. The release size was 170.0 MiB for go1.11rc1, this change reduces the size by 6.3 MiB, a 3.7% reduction. After this change, and others which reduced the release size, the release candidate is now 115.2 MiB. Updates golang/go#27151 Change-Id: I6687c34f3d4ae161c5e6df1f7af8cf3adc016fc4 Reviewed-on: https://go-review.googlesource.com/131156 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Change https://golang.org/cl/135495 mentions this issue: |
The previous book was 387 KiB decompressed and 119 KiB compressed, the new book is 567 KiB decompressed and 132 KiB compressed. Overall, this change will reduce the release binary size by 196 KiB. The new book will allow for slightly more extensive compression testing with a larger text. Command to run the benchmark tests used with benchstat: `../bin/go test -run='^$' -count=4 -bench=. compress/bzip2 compress/flate` When running the benchmarks locally, changed "Newton" to "Twain" and filtered the tests with the -bench flag to include only those which were relevant to these changes. benchstat results below: name old time/op new time/op delta DecodeTwain-8 19.6ms ± 2% 24.1ms ± 1% +23.04% (p=0.029 n=4+4) Decode/Twain/Huffman/1e4-8 140µs ± 3% 139µs ± 5% ~ (p=0.886 n=4+4) Decode/Twain/Huffman/1e5-8 1.27ms ± 3% 1.26ms ± 1% ~ (p=1.000 n=4+4) Decode/Twain/Huffman/1e6-8 12.4ms ± 0% 13.2ms ± 1% +6.42% (p=0.029 n=4+4) Decode/Twain/Speed/1e4-8 133µs ± 1% 123µs ± 1% -7.35% (p=0.029 n=4+4) Decode/Twain/Speed/1e5-8 1.20ms ± 0% 1.02ms ± 3% -15.32% (p=0.029 n=4+4) Decode/Twain/Speed/1e6-8 12.0ms ± 2% 10.1ms ± 3% -15.89% (p=0.029 n=4+4) Decode/Twain/Default/1e4-8 131µs ± 6% 108µs ± 5% -17.84% (p=0.029 n=4+4) Decode/Twain/Default/1e5-8 1.06ms ± 2% 0.80ms ± 1% -24.97% (p=0.029 n=4+4) Decode/Twain/Default/1e6-8 10.0ms ± 3% 8.0ms ± 3% -20.06% (p=0.029 n=4+4) Decode/Twain/Compression/1e4-8 128µs ± 4% 115µs ± 4% -9.70% (p=0.029 n=4+4) Decode/Twain/Compression/1e5-8 1.04ms ± 2% 0.83ms ± 4% -20.37% (p=0.029 n=4+4) Decode/Twain/Compression/1e6-8 10.4ms ± 4% 8.1ms ± 5% -22.25% (p=0.029 n=4+4) Encode/Twain/Huffman/1e4-8 55.7µs ± 2% 55.6µs ± 1% ~ (p=1.000 n=4+4) Encode/Twain/Huffman/1e5-8 441µs ± 0% 435µs ± 2% ~ (p=0.343 n=4+4) Encode/Twain/Huffman/1e6-8 4.31ms ± 4% 4.30ms ± 4% ~ (p=0.886 n=4+4) Encode/Twain/Speed/1e4-8 193µs ± 1% 166µs ± 2% -14.09% (p=0.029 n=4+4) Encode/Twain/Speed/1e5-8 1.54ms ± 1% 1.22ms ± 1% -20.53% (p=0.029 n=4+4) Encode/Twain/Speed/1e6-8 15.3ms ± 1% 12.2ms ± 3% -20.62% (p=0.029 n=4+4) Encode/Twain/Default/1e4-8 393µs ± 1% 390µs ± 1% ~ (p=0.114 n=4+4) Encode/Twain/Default/1e5-8 6.12ms ± 4% 6.02ms ± 5% ~ (p=0.486 n=4+4) Encode/Twain/Default/1e6-8 69.4ms ± 5% 59.0ms ± 4% -15.07% (p=0.029 n=4+4) Encode/Twain/Compression/1e4-8 423µs ± 2% 379µs ± 2% -10.34% (p=0.029 n=4+4) Encode/Twain/Compression/1e5-8 7.00ms ± 1% 7.88ms ± 3% +12.49% (p=0.029 n=4+4) Encode/Twain/Compression/1e6-8 76.6ms ± 5% 80.9ms ± 3% ~ (p=0.114 n=4+4) name old speed new speed delta DecodeTwain-8 19.8MB/s ± 2% 23.6MB/s ± 1% +18.84% (p=0.029 n=4+4) Decode/Twain/Huffman/1e4-8 71.7MB/s ± 3% 72.1MB/s ± 6% ~ (p=0.943 n=4+4) Decode/Twain/Huffman/1e5-8 78.8MB/s ± 3% 79.5MB/s ± 1% ~ (p=1.000 n=4+4) Decode/Twain/Huffman/1e6-8 80.5MB/s ± 0% 75.6MB/s ± 1% -6.03% (p=0.029 n=4+4) Decode/Twain/Speed/1e4-8 75.2MB/s ± 1% 81.2MB/s ± 1% +7.93% (p=0.029 n=4+4) Decode/Twain/Speed/1e5-8 83.4MB/s ± 0% 98.6MB/s ± 3% +18.16% (p=0.029 n=4+4) Decode/Twain/Speed/1e6-8 83.6MB/s ± 2% 99.5MB/s ± 3% +18.91% (p=0.029 n=4+4) Decode/Twain/Default/1e4-8 76.3MB/s ± 6% 92.8MB/s ± 4% +21.62% (p=0.029 n=4+4) Decode/Twain/Default/1e5-8 94.4MB/s ± 3% 125.7MB/s ± 1% +33.24% (p=0.029 n=4+4) Decode/Twain/Default/1e6-8 100MB/s ± 3% 125MB/s ± 3% +25.12% (p=0.029 n=4+4) Decode/Twain/Compression/1e4-8 78.4MB/s ± 4% 86.8MB/s ± 4% +10.73% (p=0.029 n=4+4) Decode/Twain/Compression/1e5-8 95.7MB/s ± 2% 120.3MB/s ± 4% +25.65% (p=0.029 n=4+4) Decode/Twain/Compression/1e6-8 96.4MB/s ± 4% 124.0MB/s ± 5% +28.64% (p=0.029 n=4+4) Encode/Twain/Huffman/1e4-8 179MB/s ± 2% 180MB/s ± 1% ~ (p=1.000 n=4+4) Encode/Twain/Huffman/1e5-8 227MB/s ± 0% 230MB/s ± 2% ~ (p=0.343 n=4+4) Encode/Twain/Huffman/1e6-8 232MB/s ± 4% 233MB/s ± 4% ~ (p=0.886 n=4+4) Encode/Twain/Speed/1e4-8 51.8MB/s ± 1% 60.4MB/s ± 2% +16.43% (p=0.029 n=4+4) Encode/Twain/Speed/1e5-8 65.1MB/s ± 1% 81.9MB/s ± 1% +25.83% (p=0.029 n=4+4) Encode/Twain/Speed/1e6-8 65.2MB/s ± 1% 82.2MB/s ± 3% +26.00% (p=0.029 n=4+4) Encode/Twain/Default/1e4-8 25.4MB/s ± 1% 25.6MB/s ± 1% ~ (p=0.114 n=4+4) Encode/Twain/Default/1e5-8 16.4MB/s ± 4% 16.6MB/s ± 5% ~ (p=0.486 n=4+4) Encode/Twain/Default/1e6-8 14.4MB/s ± 6% 17.0MB/s ± 4% +17.67% (p=0.029 n=4+4) Encode/Twain/Compression/1e4-8 23.6MB/s ± 2% 26.4MB/s ± 2% +11.54% (p=0.029 n=4+4) Encode/Twain/Compression/1e5-8 14.3MB/s ± 1% 12.7MB/s ± 3% -11.08% (p=0.029 n=4+4) Encode/Twain/Compression/1e6-8 13.1MB/s ± 4% 12.4MB/s ± 3% ~ (p=0.114 n=4+4) name old alloc/op new alloc/op delta DecodeTwain-8 3.63MB ± 0% 3.63MB ± 0% +0.15% (p=0.029 n=4+4) Decode/Twain/Huffman/1e4-8 42.0kB ± 0% 41.3kB ± 0% -1.62% (p=0.029 n=4+4) Decode/Twain/Huffman/1e5-8 43.5kB ± 0% 45.1kB ± 0% +3.74% (p=0.029 n=4+4) Decode/Twain/Huffman/1e6-8 71.7kB ± 0% 80.0kB ± 0% +11.55% (p=0.029 n=4+4) Decode/Twain/Speed/1e4-8 41.2kB ± 0% 41.3kB ± 0% ~ (p=0.286 n=4+4) Decode/Twain/Speed/1e5-8 45.1kB ± 0% 43.9kB ± 0% -2.80% (p=0.029 n=4+4) Decode/Twain/Speed/1e6-8 72.8kB ± 0% 81.3kB ± 0% +11.72% (p=0.029 n=4+4) Decode/Twain/Default/1e4-8 41.2kB ± 0% 41.2kB ± 0% -0.22% (p=0.029 n=4+4) Decode/Twain/Default/1e5-8 44.4kB ± 0% 43.0kB ± 0% -3.02% (p=0.029 n=4+4) Decode/Twain/Default/1e6-8 71.0kB ± 0% 61.8kB ± 0% -13.00% (p=0.029 n=4+4) Decode/Twain/Compression/1e4-8 41.3kB ± 0% 41.2kB ± 0% -0.29% (p=0.029 n=4+4) Decode/Twain/Compression/1e5-8 43.3kB ± 0% 43.0kB ± 0% -0.72% (p=0.029 n=4+4) Decode/Twain/Compression/1e6-8 69.1kB ± 0% 63.7kB ± 0% -7.90% (p=0.029 n=4+4) name old allocs/op new allocs/op delta DecodeTwain-8 51.0 ± 0% 51.2 ± 1% ~ (p=1.000 n=4+4) Decode/Twain/Huffman/1e4-8 15.0 ± 0% 14.0 ± 0% -6.67% (p=0.029 n=4+4) Decode/Twain/Huffman/1e5-8 20.0 ± 0% 23.0 ± 0% +15.00% (p=0.029 n=4+4) Decode/Twain/Huffman/1e6-8 134 ± 0% 161 ± 0% +20.15% (p=0.029 n=4+4) Decode/Twain/Speed/1e4-8 17.0 ± 0% 18.0 ± 0% +5.88% (p=0.029 n=4+4) Decode/Twain/Speed/1e5-8 30.0 ± 0% 31.0 ± 0% +3.33% (p=0.029 n=4+4) Decode/Twain/Speed/1e6-8 193 ± 0% 228 ± 0% +18.13% (p=0.029 n=4+4) Decode/Twain/Default/1e4-8 17.0 ± 0% 15.0 ± 0% -11.76% (p=0.029 n=4+4) Decode/Twain/Default/1e5-8 28.0 ± 0% 32.0 ± 0% +14.29% (p=0.029 n=4+4) Decode/Twain/Default/1e6-8 199 ± 0% 158 ± 0% -20.60% (p=0.029 n=4+4) Decode/Twain/Compression/1e4-8 17.0 ± 0% 15.0 ± 0% -11.76% (p=0.029 n=4+4) Decode/Twain/Compression/1e5-8 28.0 ± 0% 32.0 ± 0% +14.29% (p=0.029 n=4+4) Decode/Twain/Compression/1e6-8 196 ± 0% 150 ± 0% -23.47% (p=0.029 n=4+4) Updates #27151 Change-Id: I6c439694ed16a33bb4c63fbfb8570c7de46b4f2d Reviewed-on: https://go-review.googlesource.com/135495 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Change https://golang.org/cl/138495 mentions this issue: |
This text is used mainly for benchmark compression testing, and in one net test. The text was prevoiusly in a src/testdata directory, but since that directory would only include one file, the text is moved to the existing src/compression/testdata directory. This does not cause any change to the benchmark results. Updates #27151 Change-Id: I38ab5089dfe744189a970947d15be50ef1d48517 Reviewed-on: https://go-review.googlesource.com/138495 Run-TryBot: Katie Hockman <katie@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Change https://golang.org/cl/138737 mentions this issue: |
…ss/testdata" This reverts commit 067bb44. Reason for revert: Failing Darwin-arm builds because that testing environment does not access testdata from sibling directories. A future change will likely be made to move this testdata out of src/testdata to create a solution that doesn't require the single-file directory. Updates #27151 Change-Id: I8dbf5dd9512c94a605ee749ff4655cb00b0de686 Reviewed-on: https://go-review.googlesource.com/138737 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Change https://golang.org/cl/143537 mentions this issue: |
Now that the tour is removed from cmd/release (for Go 1.12), and CLI support is removed from godoc so it's only a webserver, is there any value in shipping the godoc webserver with releases? I think we should remove it and trim the size of releases further. Anybody wants to read the docs offline can /cc @andybons @katiehockman @ianlancetaylor @FiloSottile @bcmills |
(Braindump about whether we make "lite" releases.... I'm leaning towards no) We currently ship 15 downloads with releases at https://golang.org/dl/ ....
The binary archives take about 112MB compressed, 329MB extracted (linux-amd64 sizes). People sometimes complain that these are too large, and they've been growing over time. (See meta bug #27151) One thing we could do is ship "lite" versions without:
Maybe also:
It looks like we could halve the sizes of the downloads. (compressed & on-disk) The question is whether we want to. If we had "lite" versions, who is their intended audience? If it's CI users doing builds in loops rather than humans, then one might counter argue that they'd benefit from pre-built *.a files (for faster builds) and the bandwidth isn't a concern for a cloud-connected CI system. If it's for people building Docker images, one could counter argue that multi-stage Dockerfiles solve the problem: you should build your application in some heavy stage, then copy your final binary to a small image. |
Go for it. I have no issue as long as there's a path to having offline docs (which one can get through |
…leases The tour binary was renamed from gotour to tour in CL 141857 as part of golang/go#28163, but cmd/release wasn't updated to know about the new name. This fixes that. Additionally, don't push tour source to the buildlet if the releaselet isn't going to include it anyway. Tested that running it with a Go 1.11 release includes the tour: $ release -rev release-branch.go1.11 -tools release-branch.go1.11 \ -target=linux-amd64 -skip_tests -version=go1.11.99 -watch And with Go 1.12 does not: $ release -rev master -tools master \ -target=linux-amd64 -skip_tests -version=go1.12.99 -watch Updates golang/go#28163 Updates golang/go#27151 Change-Id: I246fc274c9dea81a24d27322bcbbe5a59e682a5e Reviewed-on: https://go-review.googlesource.com/c/143537 Reviewed-by: Katie Hockman <katie@golang.org>
Change https://golang.org/cl/144281 mentions this issue: |
Saves about 800k compressed, 2.2MB on disk. Updates golang/go#27151 Change-Id: Ie78e02c3956b81c18643c80620f97612d55fdd23 Reviewed-on: https://go-review.googlesource.com/c/144281 Reviewed-by: Andrew Bonventre <andybons@golang.org>
Per #29713, does it make sense to omit the |
@darkfeline, if there are some users who don't want to be able to run tests, yes. See discussion above about "lite" builds and who their audience might be. |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This sounds reasonable to me. There's a minor anticipated benefit of doing this. I expect that @bradfitz This hasn't happened by now. Is it too late for 1.12 now? |
This comment has been minimized.
This comment has been minimized.
Update to first comment: Go 1.12.4 linux-amd64: 122 MB |
Change https://golang.org/cl/174322 mentions this issue: |
Tested that it's still present in Go 1.12: $ release -version=go1.12.999 -target=linux-amd64 -watch -skip_tests -rev=release-branch.go1.12 -tools=release-branch.go1.12 -net=release-branch.go1.12 ... $ ls -lh go1.12.999.linux-amd64.tar.gz -rw-r--r-- 1 bradfitz bradfitz 122M Apr 29 19:28 go1.12.999.linux-amd64.tar.gz $ tar -ztf go1.12.999.linux-amd64.tar.gz | grep godoc go/bin/godoc go/src/cmd/vendor/github.com/google/pprof/profile/testdata/go.godoc.thread go/src/cmd/vendor/github.com/google/pprof/profile/testdata/go.godoc.thread.string But not with Go 1.13: $ release -version=go1.13beta0 -target=linux-amd64 -watch -skip_tests -rev=8c1f78524e421ac01e35e8805dd7a45bf98c2a79 ... $ ls -lh go1.13beta0.linux-amd64.tar.gz -rw-r--r-- 1 bradfitz bradfitz 115M Apr 29 19:00 go1.13beta0.linux-amd64.tar.gz $ tar -ztf go1.13beta0.linux-amd64.tar.gz | grep godoc $ Fixes golang/go#30029 Updates golang/go#27151 Change-Id: I9ac5c1b2bc76f184bf05fdcac86bb2e37b57c77c Reviewed-on: https://go-review.googlesource.com/c/build/+/174322 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Why not support more format downloads, for example
|
As of 1.21, releases are much smaller, in the 60-70MiB range. I think that's good enough to close this. |
Go1's source download was 11 MB.
Go 1.11rc1's source download is 20MB.
Go 1.4.3's Linux download was 58 MB.
Go 1.11rc1's Linux download was 169 MB.
Clearly 169 MB is pretty ridiculous.
https://go-review.googlesource.com/c/build/+/129955 removed some useless *.a files to save 20 MB.
https://go-review.googlesource.com/c/build/+/130575 removed the blog to save 34MB.
We're now down to 116 MB for what will be the Go 1.11rc2 Linux download. Better, but not great.
This is a tracking bug for further improvements.
/cc @ianlancetaylor @andybons @dmitshur
The text was updated successfully, but these errors were encountered: