Check in busybox to avoid historically flakey download in CI and simplify process

[LilithHafner]

...Because these downloads have been causing troubles forever and it's only 608KB

Here are some recent CI failures due to this download system:

https://buildkite.com/julialang/julia-master/builds/28231#018ae4d5-3291-4b19-be81-54b6f308bc01/12669-12883
https://buildkite.com/julialang/julia-master/builds/28230#018ae460-5e84-4ec2-9b6b-31bcfd89c3b2/12635-12891
https://buildkite.com/julialang/julia-master/builds/28222#018ae28d-ba5f-4869-b01a-de6e2690d71e/12707-13044
https://buildkite.com/julialang/julia-master/builds/28220#018ae26b-1847-4a28-bee0-1a6877b0ee69/12578-12847
https://buildkite.com/julialang/julia-master/builds/28213#018ae18a-db87-44c9-b149-99302322076c/12703-12992
https://buildkite.com/julialang/julia-master/builds/28209#018ae092-b991-48ee-982a-32fb0f29719a/12475-12785

[LilithHafner]

(this is totally untested because I don't have a windows machine so I'm using CI to debug)

[LilithHafner]

I have not reviewed the Binary file I'm checking in. However, I do attest that at the time of this PR, the binary I'm checking in is the same as a file I downloaded from https://frippery.org/files/busybox/busybox.exe and the same as a file I downloaded from https://web.archive.org/web/20230828161044/https://frippery.org/files/busybox/busybox.exe all of which have a sha512 checksum of a5b7c95f1aa9caeff134557f18070ec58c352f78ae09e3b9cc379547ff9278b207395ce62d9b73ab194f4d865bbd7f6875eb35336bfb9989b0594f493a443cbf
Hopefully we trust frippery.org, because we've been automatically downloading and executing binaries from them during CI for a while now.

[IanButterworth]

See #51531

[DilumAluthge]

I don't think we should check binary .exe files into this repo.

If we really need to, here's what we can do instead:

1. Create a new GitHub repo that just contains a README.
2. Upload the BusyBox files as GitHub Releases on that repo.

[LilithHafner]

@DilumAluthge, why not? It seems like a fixed binary is at least as secure and auditable as a bit of code that downloads and runs a binary from a remote source.

[IanButterworth]

I think we can close this?

[LilithHafner]

I'm still not sure why it's so bad to check in a binary file rather than download it each time. Would one of you explain the reasoning behind that?

[IanButterworth]

Perhaps this should be reopened with a different title? I've got no stance on the question you're asking

[DilumAluthge]

It's generally recommended to only check text files into Git. For binary files, people usually use Git LFS or some other solution.

[IanButterworth]

For the security concern we could add a hash check of the downloaded file?

[DilumAluthge]

Yeah, if there are security concerns, checksums are the way to go. We already do that for all our build dependencies, so it seems reasonable to add checksums for our various test dependencies as well.

[LilithHafner]

Perhaps this should be reopened with a different title?

Okay, done.

It's generally recommended to only check text files into Git.

Who recommends this and why?

For binary files, people usually use Git LFS or some other solution.

I assume that Git LFS is appropriate for storing large files. is 608 KB too large?

[LilithHafner]

@DilumAluthge, I appreciate that you block PRs that should not be merged. However, I think it is appropriate to provide an explanation of why you choose to block such a PR and engage in a brief dialogue about it. I still don't understand why it is bad to check in a small binary file.

[gbaraldi]

Because it gets added to the julia repo, and it doesn't make sense to be there. Every git clone will download it.

[DilumAluthge]

Just a couple quick quotes:

Unfortunately git does not handle binary files elegantly (unless you use git-lfs). You can inflate storage rapidly by, say, editing a 10M zip file a few times. I've had to GC more than one repo where someone accidentally added an innocuous binary file, and the next thing you know the repo has exceeded 2G of storage space.

(Source)

---

It’s important to never commit binary files because once you’ve commit them they are in the repository history and are very annoying to remove. You can delete the files from the current version of the project - but they’ll remain in the repository history, meaning that the overall repository size will still be large.

(Source)

[vchuravy]

Yeah checking binaries is a no-go. We can try and address this on the CI side of things (provide the binary in the environment there instead of downloading it everytime),
but checking in a binary should be a measure of last resort, if at all.

[LilithHafner]

Thanks, all, for the explanations. I appreciate understanding the reasoning behind this. IIUC the issue is all about repo file size bloat.