Fix slow move on same object bucket during file upload. #3205
Conversation
This commit fixes the issue nextcloud/server#47856. When you upload a file into a group folder and when you use a single S3 bucket as primary storage, the final move operation hangs for a long time. In the background, Nextcloud initiates a copy-delete sequence from the bucket into the bucket, with causes a lot unnecessary overhead. Nextcloud thinks that the file must be imported to another storage and does not recognize that everything is done on the same object bucket. In that case, the import step can be completely skipped, which saves time, network bandwidth and reduces the load on the object storage. Signed-off-by: Christoph Fiehe <c.fiehe@eurodata.de>
susnux
added
enhancement
3. to review
There was a problem hiding this comment.
Looks good, but would wait for @icewind1991 opinion. Maybe this logic should be moved to the server repo.
cfiehe
force-pushed
the
fix_slow_move_on_same_bucket
branch
from
1b9a8cb to
1faaaca
Compare
I'm pretty sure this is the case. It's also something I suggested in nextcloud/server#47682 as I noticed the same problem when you only upload a single (large) chunk. |
Agreed. Doing the comparison with the `===` operator is better, because it is stricter. Thank you very much for the suggestion. Co-authored-by: Ferdinand Thiessen <opensource@fthiessen.de> Signed-off-by: cfiehe <cfiehe@users.noreply.github.com>
cfiehe
force-pushed
the
fix_slow_move_on_same_bucket
branch
from
1faaaca to
e7b6f2c
Compare
|
Well, I think the fix can stay in the plugin until Nextcloud has developed further and handles file renaming on the same storage in a more efficient way than today. Unfortunately, I cannot find something similar in case of a local storage. Uploads in a group folder on a local storage always cause an inefficient copy-delete on Nextcloud triggered by the |
|
Nice PR! Let's wait indeed for @icewind1991. If everything is ok, I'd say we merge this one as it would be a significant performance boost. We can then STILL move the logic to server afterwards :) |
|
This logic should probably go into Also instead of doing nothing/ |
|
@icewind1991 @provokateurin |
|
Closing this in favor of the server PR :) |
|
Hello there, We hope that the review process is going smooth and is helpful for you. We want to ensure your pull request is reviewed to your satisfaction. If you have a moment, our community management team would very much appreciate your feedback on your experience with this PR review process. Your feedback is valuable to us as we continuously strive to improve our community developer experience. Please take a moment to complete our short survey by clicking on the following link: https://cloud.nextcloud.com/apps/forms/s/i9Ago4EQRZ7TWxjfmeEpPkf6 Thank you for contributing to Nextcloud and we hope to hear from you soon! (If you believe you should not receive this message, you can add yourself to the blocklist.) |
This commit fixes the issue nextcloud/server#47856. When you upload a file into a group folder and when you use a single S3 bucket as primary storage, the final move operation hangs for a long time. In the background, Nextcloud initiates a copy-delete sequence from the bucket into the bucket, with causes a lot unnecessary overhead. Nextcloud thinks that the file must be imported to another storage and does not recognize that everything is done on the same object bucket. In that case, the import step can be completely skipped, which saves time, network bandwidth and reduces the load on the object storage.