-
-
Notifications
You must be signed in to change notification settings - Fork 30.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
explore hashlib use of the Linux Kernel CryptoAPI #91258
Comments
Linux kernels provide a CryptoAPI. This is a common place for platform specific hardware accelerated hash algorithms to be exposed to the user (especially on SoCs which often have non-standard hardware). https://www.kernel.org/doc/html/v4.10/crypto/userspace-if.html hashlib currently uses OpenSSL when possible for performance. We could also look at querying the kernel API. How to decide between the two implementations when both are present is something TBD. This would probably be best done via a configure time check for libkcapi? |
We don't need libkcapi. I added AF_ALG support a while ago: import binascii
import os
import socket
with socket.socket(socket.AF_ALG, socket.SOCK_SEQPACKET, 0) as cfgsock:
cfgsock.bind(("hash", "sha256"))
opsock, _ = cfgsock.accept()
with opsock:
with open("/etc/os-release") as f:
st = os.fstat(f.fileno())
# blindly assumes that sendfile() exhausts the fd.
os.sendfile(
opsock.fileno(), f.fileno(), offset=0, count=st.st_size
)
res = opsock.recv(512)
print(binascii.hexlify(res)) |
Neat. I've never used the API, just filing a breadcrumb suggesting we see if it makes sense. Being I/O based, that even takes care of GIL releasing from Python. :) |
test_socket has examples for HMAC, AES-CBC, and AES-GCM. with self.create_alg('hash', 'hmac(sha1)') as algo:
algo.setsockopt(socket.SOL_ALG, socket.ALG_SET_KEY, b"Jefe")
op, _ = algo.accept()
with op:
op.sendall(b"what do ya want for nothing?")
self.assertEqual(op.recv(512), expected) |
And sendfile() is zero-copy. Data does not have to leave Kernel space. |
I figured out how to implement copy(). dup() does not work as expected, but accept() on an AF_ALG client socket creates an independent copy. |
Playing with your draft PR branch and testing things out... the performance of the Linux kernel crypto API for hash algorithms is several times slower on a modern AMD Zen, regardless of data size. I realize it probably gets a tad better in limited situations where sendfile or splice can used, but it doesn't seem like a useful API on modern mainstream hardware. It could still make sense on low performance things that have non-userspace accelerator hardware. But this doesn't leave much motivation to implement code using it. This one sounds like it could be a bit faster https://github.com/cryptodev-linux/cryptodev-linux (and is what would be needed for this on FreeBSD and OpenBSD), but is not in mainline in Linux. |
Probably, this should be added into the documentation if this issue gets rejected. Otherwise, there will be another attempts to introduce CryptoAPI, and one of them can even accidentally slip through. |
I'm not worried about that. This issue existing in our issue tracker is sufficient documentation. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: