Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch from outdated SharpZipLib.Patched to newer SharpZipLib #3329

Merged
merged 2 commits into from
Mar 21, 2021

Conversation

DasSkelett
Copy link
Member

@DasSkelett DasSkelett commented Mar 21, 2021

Motivation

https://github.com/KSP-CKAN/NetKAN/issues/8425 and KSP-CKAN/NetKAN#8315 show some weird errors when trying to extract certain zips with special characters on Windows systems with CJK locales.
Maybe this is a bug of SharpZipLib, and maybe it is already fixed in a newer versions. It's a total stab in the dark, but worth a try. I've asked affected users to try a debug build with an updated version of the package, we should know when we get an answer.

In any case, there have been some improvements to that library in the releases since, the usual performance enhancements, and support for BZip2 compression.

We have been using a "Patched" version of this library, which has been deprecated now. Its NuGet page doesn't give any details on what it's patching, but it seems to be related to:

Changes

Switch to the upstream SharpZipLib, v1.3.1. I haven't seen any breaking changes, and mod installation and inflation still works perfectly fine in my testing.

Add venv/ folders to .gitignore - I have one inside bin/ for the Python scripts there.

Add exitstatus to bin/requirements.txt. For some reason I didn't catch the missing dependency during the PRs for the ckan-merge-pr.py.

(Gonna give this the "Enhancement" flair for the time being and not say it closes the issue https://github.com/KSP-CKAN/NetKAN/issues/8425, since we don't know for sure yet)

@DasSkelett DasSkelett added Enhancement New features or functionality Core (ckan.dll) Issues affecting the core part of CKAN Pull request Build Issues affecting the build system Netkan Issues affecting the netkan data Tests Issues affecting the internal tests labels Mar 21, 2021
@HebaruSan

This comment has been minimized.

@DasSkelett
Copy link
Member Author

Ugh, I've paid attention that it doesn't underline it in the PR preview, but apparently it only happens once the PR is created. Let's do some trickery.

Copy link
Member

@HebaruSan HebaruSan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎲

@DasSkelett DasSkelett merged commit bf7064f into KSP-CKAN:master Mar 21, 2021
@DasSkelett DasSkelett deleted the fix/update-sharpziplib branch March 21, 2021 21:26
@HebaruSan
Copy link
Member

Hmm, this might have caused a new problem:

image

That mod hasn't updated since August, and it doesn't contain any duplicates:

image

Is this version of the lib less able to handle CJK locales?

@DasSkelett
Copy link
Member Author

Interestingly this inflates and installs successfully for me, even though I had problems with UTF-8 in #3323 😕

@HebaruSan
Copy link
Member

Interestingly this inflates and installs successfully for me

Me, too. Maybe the Inflator's instance is missing some locale data?

@DasSkelett
Copy link
Member Author

Now this is interesting. Running mono netkan.exe Wanhu-Common.netkan inside a mono:latest container:

root@3f119569761c:/ckan/_build# mono netkan.exe --version
v1.30.1+bf7064fbd8fa

root@3f119569761c:/ckan/_build# mono netkan.exe Wanhu-Common.netkan
5855 [1] FATAL CKAN.NetKAN.Program (null) - https://spacedock.info/mod/2507/LongMarch-10?CZ-10?/download/1.3 is not a valid ZIP file:
Error instep EntryHeader for Wanhu/Wanhu_Parts/Flags/Flag??????.png: Exception during test - 'Name is invalid'

I expected to see the "multiple files" error, but instead got the on from https://github.com/KSP-CKAN/NetKAN/issues/8425, but on Linux 🤯

@techman83
Copy link
Member

Good chance the container is missing some locale data. I doubt it's included.

@DasSkelett
Copy link
Member Author

Looks like SharpZipLib changed the default for reading and writing zips away from UTF-8 in version 1.1.0:
icsharpcode/SharpZipLib#591

Setting ICSharpCode.SharpZipLib.Zip.ZipStrings.UseUnicode = true; right at the beginning in Program.cs solves the above exception and makes netkan.exe work in a completely fresh Mono container again 🎉

@HebaruSan
Copy link
Member

HebaruSan commented Mar 29, 2021

Setting ICSharpCode.SharpZipLib.Zip.ZipStrings.UseUnicode = true; right at the beginning

Is there a non-static equivalent? That's a bit ugly in terms of encapsulation.

@DasSkelett
Copy link
Member Author

There seems to be an overhaul of this in progress, which also mentions going back to support Unicode by default (for creating new zips judging from the discussion, not sure if and how this would affect reading zips):
icsharpcode/SharpZipLib#592

Otherwise I haven't really digged into it yet, I'm currently trying to write a unit test for this in order to prevent this in the future, and also simpler patch testing.

@HebaruSan
Copy link
Member

It looks like the SharpZipLib team is moving in a good direction, but it's less clear when those changes will land. I'm comfortable setting the static bool as a short term workaround, and I've subscribed to their repo's release notifications so we can revisit this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Build Issues affecting the build system Core (ckan.dll) Issues affecting the core part of CKAN Enhancement New features or functionality Netkan Issues affecting the netkan data Tests Issues affecting the internal tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants