Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Git repository bloat from precompiled binaries #241

Open
dylanahsmith opened this issue Nov 29, 2021 · 0 comments
Open

Git repository bloat from precompiled binaries #241

dylanahsmith opened this issue Nov 29, 2021 · 0 comments

Comments

@dylanahsmith
Copy link
Collaborator

As part of #165 we will be changing the V8 precompiled binary more often. Despite removing the windows precompiled binary for now (#234), we have also been adding more precompiled binaries for ARM64 (#202 & #223).

These sizes for these binaries are:

  • 29M for darwin_arm64/libv8.a
  • 32M for darwin_x86_64/libv8.a
  • 49M for linux_x86_64/libv8.a

The documentation about large files on GitHub says

GitHub limits the size of files allowed in repositories. If you attempt to add or update a file that is larger than 50 MB, you will receive a warning from Git. The changes will still successfully push to your repository, but you can consider removing the commit to minimize performance impact.

Git isn't really designed well for managing large files and wasn't intended for use for storing precompiled binaries. The default behaviour for checking out a file will checkout all the data that was ever committed and to keep any referenced commits, so these large files can lead to slow checkouts. This additional load affects Github, as well as the git client checkout out the code, which seems to be why Github has limits in place to help protect their service. Instead, it is generally recommended to release precompiled binaries separately, which is what Github's releases features is intended for.

v8go is imported from rogchap.com/v8go. Go documents remote import paths special casing how code is imported from Github, which I presume allows it to leverage Github releases, which doesn't seem like it would be used for v8go since it has some indirection to avoid using github.com/rogchap/v8go as the import path. Instead, Go makes a request to https://rogchap.com/v8go which serves https://github.com/rogchap/rogchap.github.io/blob/master/v8go/index.html. Go will find the <meta name="go-import" content="rogchap.com/v8go git https://github.com/rogchap/v8go"> tag which will tell it to checkout the code using git. That Go documentation page also says

When using modules, an additional variant of the go-import meta tag is recognized and is preferred over those listing version control systems. That variant uses "mod" as the vcs in the content value, as in:

<meta name="go-import" content="example.org mod https://code.org/moduleproxy">

This tag means to fetch modules with paths beginning with example.org from the module proxy available at the URL https://code.org/moduleproxy. See https://golang.org/ref/mod#goproxy-protocol for details about the proxy protocol.

That protocol will download the module as a zip file, so could be used to serve a Github release that includes a precompiled V8 library.

However, it is convenient to be able to try out the latest changes in v8go before a release is made, but using Git would preclude downloading any precompiled binaries that were moved to releases. This inconvenience could be avoided if we extracted the precompiled v8 library packages (deps/{darwin_arm64,darwin_arm64,linux_x86_64}) into their own repo as a separate Go module, so the binary release for it could be downloaded while still using a v8go version from Git. This separation would also avoid this inconvenience for typical development of v8go itself, continuing to avoid the need to compile libv8 in order to test the code.

@dylanahsmith dylanahsmith changed the title Git repository Bloat from precompiled binaries Git repository bloat from precompiled binaries Nov 29, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant