Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase SFTP chunk size to increase the SFTP throughput in both directions #664

Open
wants to merge 10 commits into
base: devel
Choose a base branch
from

Conversation

Jakuje
Copy link
Contributor

@Jakuje Jakuje commented Nov 19, 2024

SUMMARY

This is built on top of #638 so it depends on that change

Previously, the 1024b chunk was used, which resulted in large traffic overhead as each of the chunk needs to be wrapped in SFTP packet, as well as large overhead on high-latency links as the SFTP is now synchronous, waiting for every packet confirmation.

The libssh documentation recommends to use 16kB chunks, but using 32kB chunks should be safe as it still fits to the recommended limits in the SFTP specification. Using large reads could cause issues such as curl/curl#11804

ISSUE TYPE
  • Bugfix Pull Request

@psf-chronographer psf-chronographer bot added the bot:chronographer:provided There is a change note present in this PR label Nov 19, 2024
@Jakuje Jakuje mentioned this pull request Nov 19, 2024
Copy link

Congratulations! One of the builds has completed. 🍾

You can install the built RPMs by following these steps:

  • sudo yum install -y dnf-plugins-core on RHEL 8
  • sudo dnf install -y dnf-plugins-core on Fedora
  • dnf copr enable packit/ansible-pylibssh-664
  • And now you can install the packages.

Please note that the RPMs should be used only in a testing environment.

Fixes: ansible#341
Signed-off-by: Jakub Jelen <jjelen@redhat.com>
Signed-off-by: Jakub Jelen <jjelen@redhat.com>
Signed-off-by: Jakub Jelen <jjelen@redhat.com>
Thanks @kucharskim for the report and testing!

Signed-off-by: Jakub Jelen <jjelen@redhat.com>
Signed-off-by: Jakub Jelen <jjelen@redhat.com>
Signed-off-by: Jakub Jelen <jjelen@redhat.com>
Signed-off-by: Jakub Jelen <jjelen@redhat.com>
src/pylibsshext/sftp.pyx Outdated Show resolved Hide resolved
Signed-off-by: Jakub Jelen <jjelen@redhat.com>
Signed-off-by: Jakub Jelen <jjelen@redhat.com>
Signed-off-by: Jakub Jelen <jjelen@redhat.com>
Copy link

@Jakuje
Copy link
Contributor Author

Jakuje commented Dec 22, 2024

The 🐍 3.9@macos-13 job is weird. It fails the first time (with some internal error -- not sure if we can do anything about that), but works on maximum verbosity ...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bot:chronographer:provided There is a change note present in this PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants