SSL related leak

[rgs1]

We are seeing a memory leak when using the following tls_context config:

   "tls_context": {
      "common_tls_context": {
        "tls_certificate_sds_secret_configs": [
          {
            "name": "certificate_chain",
            "sds_config": {
              "api_config_source": {
                "api_type": "GRPC",
                "grpc_services": [
                  {
                    "envoy_grpc": {
                      "cluster_name": "local_sds"
                    }
                  }
                ]
              },
              "initial_fetch_timeout": "10s"
            }
          }
        ],
        "validation_context_sds_secret_config": {
          "name": "foo",
          "sds_config": {
            "api_config_source": {
              "api_type": "GRPC",
              "grpc_services": [
                {
                  "envoy_grpc": {
                    "cluster_name": "local_sds"
                  }
                }
              ]
            },
            "initial_fetch_timeout": "10s"
          }
        }
      }

Per pprof:

Total: 123.1 MB                                                                                                                                            
    98.9  80.3%  80.3%     98.9  80.3% OPENSSL_malloc                                                                                                      
    10.5   8.5%  88.9%     10.5   8.5% OPENSSL_realloc                                                                                                     
     1.3   1.1%  89.9%      2.2   1.8% Envoy::Event::DispatcherImpl::createClientConnection                                                                
     1.2   0.9%  90.9%      1.8   1.4% Envoy::Event::DispatcherImpl::createServerConnection                    
     0.8   0.7%  91.6%      1.2   1.0% Envoy::Http::CodecClientProd::CodecClientProd                                 
     0.8   0.7%  92.2%      0.8   0.7% Envoy::Buffer::OwnedImpl::reserve                                 
     0.8   0.6%  92.9%      0.8   0.6% std::make_unique@8e1970
...

This still happens with master as of yesterday, but apparently started a week or so ago. I'll try to bisect and see if any of the recent BoringSSL updates might be related.

cc: @fishcakez @derekargueta

[rgs1]

Possible leak per pprof:

994      (0000000000ac0112) ??:0:OPENSSL_malloc
         (0000000000a46555) ??:0:asn1_enc_save
         (0000000000a43a26) ??:0:asn1_item_ex_d2i
         (0000000000a447d2) ??:0:asn1_template_noexp_d2i
         (0000000000a43e58) ??:0:asn1_template_ex_d2i
         (0000000000a436cb) ??:0:asn1_item_ex_d2i
         (0000000000a42fd2) ??:0:ASN1_item_d2i
         (0000000000ac1312) ??:0:parse_x509
         (0000000000ac1058) ??:0:PEM_X509_INFO_read_bio
         (000000000063d832) ??:0:Envoy::Extensions::TransportSockets::Tls::ContextImpl::ContextImpl
         (0000000000642af5) ??:0:Envoy::Extensions::TransportSockets::Tls::ClientContextImpl::ClientContextImpl
         (000000000064ad36) ??:0:Envoy::Extensions::TransportSockets::Tls::ContextManagerImpl::createSslClientContext
         (0000000000635d3a) ??:0:Envoy::Extensions::TransportSockets::Tls::ClientSslSocketFactory::onAddOrUpdateSecret
         (000000000063cd97) ??:0:std::_Function_handler::_M_invoke
         (00000000006d74f0) ??:0:Envoy::Secret::SdsApi::onConfigUpdate
         (0000000000850d49) ??:0:Envoy::Config::GrpcMuxSubscriptionImpl::onConfigUpdate
         (000000000084d999) ??:0:Envoy::Config::GrpcMuxImpl::onDiscoveryResponse
         (000000000084ea52) ??:0:Envoy::Grpc::AsyncStreamCallbacks::onReceiveMessageRaw
         (00000000008765c6) ??:0:Envoy::Grpc::AsyncStreamImpl::onData
         (000000000087adf3) ??:0:Envoy::Http::AsyncStreamImpl::encodeData
         (0000000000888697) ??:0:Envoy::Router::Filter::UpstreamRequest::decodeData
         (00000000008ef728) ??:0:Envoy::Http::Http2::ConnectionImpl::onFrameReceived
         (000000000098482e) ??:0:nghttp2_session_on_data_received
         (0000000000986506) ??:0:nghttp2_session_mem_recv
         (00000000008eede8) ??:0:Envoy::Http::Http2::ConnectionImpl::dispatch
         (000000000086afe5) ??:0:Envoy::Http::CodecClient::onData
         (000000000086ba8c) ??:0:Envoy::Http::CodecClient::CodecReadFilter::onData
         (00000000006c1a80) ??:0:Envoy::Network::FilterManagerImpl::onContinueReading
         (00000000006be0eb) ??:0:Envoy::Network::ConnectionImpl::onReadReady
         (00000000006bdbb0) ??:0:Envoy::Network::ConnectionImpl::onFileEvent
         (00000000006b8a24) ??:0:Envoy::Event::FileEventImpl::assignEvents::$_0::__invoke
         (00000000009c2fdc) ??:0:event_process_active_single_queue

[htuch]

This looks like it's on the config side during context creation. @PiotrSikora any thoughts?

[rgs1]

Yep, I am trying to confirm it's actually on the config side. On the data side I am also slightly suspicious of #6326.

[lizan]

Also possibly due to recent BoringSSL upgrade: #7952

[ipuustin]

@rgs1: I looked at the #6326 changes related to this, but I did not see anything too suspicious. Please tell where the git bisect ends. My first thought was that X509_up_ref() at context_impl.cc:117 might be too much, but it has been there for a long time.

[rgs1]

So, this was just a badly configured connection pool with a huge upstream.. Sorry for the false alarm.