Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set default maxChunkSize to 99 MB #5813

Closed
wants to merge 1 commit into from
Closed

Set default maxChunkSize to 99 MB #5813

wants to merge 1 commit into from

Conversation

JustArchi
Copy link

@JustArchi JustArchi commented Jun 14, 2023

This PR:

Rationale: PR #3600 was suggested since the old value of 100 MB was arguably "not enough for modern usage". I disagree with that statement entirely, majority of people do not have ISP-grade upload speeds at home, and the default option should be adjusted accordingly to majority of users rather than some very specific setups where chunking could be a bottleneck. I do agree that chunking could result in bottleneck, and for those handful amount of setups, it's already possible to set max chunking size higher. However, there is a very good reason why chunking is for example set only to 80 MB by default in iOS app, and why it exists in general - so uploads can be retried if disrupted. 100 MB of value aligns with Cloudflare default maximum body, and I'd say there is a very good reason why they set it like that. This is why this PR sets the upper limit to 99 MB, extra megabyte just to make sure that we do not exceed the limit in regards to other data we send, such as the request URL, headers (including authorization), among others.

It's nice to mention that there is PR #4826 which addresses this issue in even better (in my opinion) way. However, that PR is in limbo state since almost a year, requires additional code review, perhaps further fixes, while a lot of people are suffering from this, being forced to change default values sometimes on even hundreds of different machines, purely because the default value was set too high for cloudflare-specific (and probably not only) usage.

Personally as unrelated developer who stumbled upon #4278 and was affected by it, I believe we should go back from 1000 MB to 100 (99) MB at least for time being, until there is better solution like #4826, without having another year or 10 with people being forced to set every single desktop nextcloud client purely because the hosting owner decided to use Cloudflare or other CDN in front of it. I'd rather have slower but working speeds for everybody by default, rather than only 82% of setups working at all, at least according to general statistics such as https://w3techs.com/technologies/details/cn-cloudflare

Cloudflare is used by 76.1% of all the websites whose reverse proxy service we know. This is 18.0% of all websites.

I understand that this can be controversial decision, that Nextcloud should not be limited by artbitrary limits set by third-party CDNs and so on and so on, I agree, but it's not like this change stops anybody from doing anything, rather, it fixes the issue that affects a lot of people using Cloudflare (among other) CDNs, and I'm strongly against software that expects from me to fix it before I'm even able to use it. If that was server setting, I'd have another opinion - if server owner decided to use Cloudflare, he can also notify the clients about the limit, simple. However, this isn't server setting, we can't tell clients what value should be used, and the burden of adapting all the clients to match it is in my opinion not worth increasing performance in very specific setups that do not handle chunking as effectively as they could.

One potential improvement is to find out the limit dynamically like in #4826. Another is to introduce server-sided max chunk size limit and use MIN(server, client) as the final one. Both of those however require knowledge and more work than this PR, I don't have enough of time to dig into it myself either, so I understand other people, this is why I believe we should for now just change the defaults, and put potential enhancements as a wishlist for when somebody is willing to improve nextcloud in this aspect.

Thank you in advance for considering this PR.

Signed-off-by: Archi <JustArchi@JustArchi.net>
Copy link
Contributor

@szaimen szaimen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would say #4826 is the better approach

@nextcloud-desktop-bot
Copy link

AppImage file: nextcloud-PR-5813-499a4a512fdf94f3d5b012109f5aee4d62767d7f-x86_64.AppImage

To test this change/fix you can simply download above AppImage file and test it.

Please make sure to quit your existing Nextcloud app and backup your data.

Copy link
Collaborator

@claucambra claucambra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have to agree with @szaimen here -- regardless, this will all require changes to incorporate support for #5819

@JustArchi
Copy link
Author

🤷

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature: 🔄 sync engine hotspot: chunk sizing Chunk sizing defaults, fallback behavior, etc.
Projects
None yet
5 participants