crypto: fix memory leak, reduce heap allocations, clean up #14122

bnoordhuis · 2017-07-07T13:36:15Z

Also of note:

The EVP_CIPHER can be reconstructed from the EVP_CIPHER_CTX instance,
no need to store it separately.

This brought to light the somewhat dubious practice of accessing the
EVP_CIPHER after the EVP_CIPHER_CTX instance had been destroyed.

It's mostly harmless due to the static nature of built-in EVP_CIPHER
instances but it segfaults when the cipher is provided by an ENGINE
and the ENGINE is unloaded because its reference count drops to zero.

I'll clean up the commit logs some more before landing.

CI: https://ci.nodejs.org/job/node-test-pull-request/9027/

bnoordhuis · 2017-07-10T11:00:27Z

I added a commit that fixes the DH memory leak reported in #8377.

CI: https://ci.nodejs.org/job/node-test-pull-request/9056/

jasnell

Largely rubber-stamp LGTM. Looked through the code and didn't see anything that stood out but didn't do a thorough review.

bnoordhuis · 2017-07-12T16:55:42Z

Bumped DH key size in test to 1024 to pacify FIPS. New CI: https://ci.nodejs.org/job/node-test-pull-request/9094/

Don't allocate + copy + free; allocate and fill in place, then hand off the pointer to Buffer::New(). Avoids unnecessary heap allocations in the following methods: - crypto.Cipher#final() - crypto.Cipher#update() - crypto.Cipheriv#final() - crypto.Cipheriv#update() - crypto.Decipher#final() - crypto.Decipher#update() - crypto.Decipheriv#final() - crypto.Decipheriv#update() - crypto.privateDecrypt() - crypto.privateEncrypt() - crypto.publicDecrypt() - crypto.publicEncrypt()

Put the 8 kB initial buffer on the stack first and don't copy it to the heap until its exact size is known (which is normally much smaller.)

Fix a memory leak by removing the heap allocation altogether. Fixes: nodejs#13917

The EVP_CIPHER can be reconstructed from the EVP_CIPHER_CTX instance, no need to store it separately. This brought to light the somewhat dubious practice of accessing the EVP_CIPHER after the EVP_CIPHER_CTX instance had been destroyed. It's mostly harmless due to the static nature of built-in EVP_CIPHER instances but it segfaults when the cipher is provided by an ENGINE and the ENGINE is unloaded because its reference count drops to zero.

The cipher kind doesn't change over the lifetime of the cipher so make it const.

Don't allocate + copy + free; allocate and fill in place, then hand off the pointer to Buffer::New(). Avoids unnecessary heap allocations in the following methods: - crypto.CryptoStream#getSession() - tls.TLSSocket#getSession()

Add a test that ensures the second call to .digest() returns an empty HMAC, like it did before. No comment on whether that is the right behavior or not.

Replace allocate + Encode() + free patterns by calls to Malloc + the Buffer::New() overload that takes ownership of the pointer. Avoids unnecessary heap allocations and copying around of data. DRY the accessor functions for the prime, generator, public key and private key properties; deletes about 40 lines of quadruplicated code.

This also renames a misnamed variable `error_` to `success_`.

Fix a memory leak in dh.setPublicKey() and dh.setPrivateKey() where the old keys weren't freed. Fixes: nodejs#8377

bnoordhuis · 2017-07-17T11:29:50Z

One more rebase + CI: https://ci.nodejs.org/job/node-test-pull-request/9192/

bnoordhuis · 2017-07-17T12:11:23Z

Stress test to see if the async-hooks/test-tlswrap failure on the osx1010 bot is a flake or not: https://ci.nodejs.org/job/node-stress-single-test/1319/