Mention caveats about exporting for Windows with embedded PCK and/or UPX

[xix-xeaon]

Bugsquad edit: Moved to a docs issue. See #3093 (comment)

---

Godot version:
Godot_v3.2-beta5_x11.64

OS/device including version:
Kubuntu/19.10

Issue description:
As recommended by reduz in godotengine/godot#27140 I used UPX to compress the executable (and the difference is so dramatic (40mb to 14mb: 65% reduction) it should really be a standard feature in my opinion) but it does not work with the Embed Pck option.

Exporting with Embed Pck for Linux/X11 gives an error but works anyway (the same as godotengine/godot#32513). But after using UPX running the executable only gives the dialog "Error: Could not load game data at path '.'. Is the .pck file missing?"

Doing the same for Windows Desktop has no error on export (and runs fine on my machine using Wine (not affected by godotengine/godot#33466), but when running UPX I get "upx: Articore.exe: CantPackException: superfluous data between sections (try --force)". Using --force leads to the same result as for the Linux/X11 version.

In godotengine/godot#18404 dns says "Godot packing/embed it the wrong way" but closed the issue since Godot 3 no longer had the Embed Pck option at that point, I don't know if it was re-implemented the same way.

Steps to reproduce:
New Project, Export, Embed Pck, UPX

[Calinou]

Consider distributing compressed releases (using a ZIP or .tar.xz archive) instead of using UPX. This avoids the many pitfalls of UPX, such as false positives in antivirus programs. This also decreases the program startup time as it no longer has to be decompressed on every startup.

[RandomShaper]

I think there's no possible way of fixing this at the Godot side. The PCK section is not consumed from memory but from disk so UPX compression renders it unreadable for Godot.

Making Godot understand a UPX-compresed file could technically work, but with the big downside of having to decompress files on the go.

Is there a way to tell UPX to keep a section uncompressed? If that's possible and UPX can keep the PCK section at the end of the file, you could at least keep the executable portion compressed.

On top of that, the PCK could contain the resources in compressed form.

[xix-xeaon]

What about LZ4 and others? In that case compression should actually speed up the loading process, even on an SSD. I just tried compressing the binary with LZ4_HC (-9) and got down to 17mb (58% reduction). If that was built into the executable it would make a big difference without having to use any zip files etc.

Only concern would be interference with stronger compression. It seems ZIP (-9) is able to pull the original down to 14mb, and ZIP on the LZ4 almost gets there as well. But XZ gets to 10mb from the original and only to 13mb from the LZ4. However, I would say LZ4 (or similar) would be a sane default with the option of turning it off.

[Calinou]

@xix-xeaon From my testing, you can get better-compressed ZIPs using 7-zip (p7zip package on Linux) with the following options:

7z a -mx9 <file>.zip <file>

This should get you a smaller ZIP compared to zip -r9 <file>.zip <file>. It still uses the DEFLATE algorithm, so it should be compatible with any decompression program.

There's another upside to distributing programs in compressed archives: on Linux, permissions can be preserved in archives, but not in "direct" file downloads. This means the user won't have to use chmod +x to make the binary executable before running it.

[girng]

FWIW, I was having issues with UPX when I UPX'd my exported game.exe. I found UPXing the godot.windows.opt.64.exe first, then exporting the game was the solution

[xix-xeaon]

@Calinou That's interesting, but in this case using 7z to ZIP only reduced the Godot binary about 400kb compared to ZIP (14mb), while using 7z with DEFLATE got it down to 11mb. 7z has a few other fun algorithms too; LZMA yields the same result as XZ (10mb, which is expected), PPMd goes to 9mb and LZMA2 goes down to 8mb. Going through LZ4 first prevents most of those gains.

[akien-mga]

Moving this to the documentation - this cannot be fixed in the engine, so there should be a warning about not using UPX to compress executables with embedded PCK.

In general, I would advise against using UPX in any situation as it's unreliable. It's simpler to just zip your game if you really want to save on download size - installed size for a 30 MB binary once unzipped should not matter.

[akien-mga]

The warning should likely go there: https://docs.godotengine.org/en/latest/development/compiling/optimizing_for_size.html#using-upx-to-compress-binaries

IMO it might be better to just drop this section too, but YMMV.

[akien-mga]

The page about exporting for Windows should have a section about embedding PCK and the caveats of that option.

The bugs in godotengine/godot#33466 and godotengine/godot#32310 should also be mentioned there (it should hopefully be fixable, but won't be fixed in 3.2).