tls.verify_hostname seems not working in [OUTPUT] forward

[duj4]

Bug Report

Describe the bug
In [OUTPUT] forward, when I enabled tls.verify_hostname, it seems this flag does not work, but in [INPUT] it does work.

Here is the basic config and which is working as expected:

centos8-1:

[OUTPUT]
    Name                forward
    Match               *
    Host                centos8-2
    tls                 on
    tls.verify          on
    tls.verify_hostname on
    tls.debug           1
    tls.ca_file         ca.crt
    tls.crt_file        centos8-1.crt
    tls.key_file        centos8-1.key
    self_hostname       ${HOSTNAME}

centos8-2:

[INPUT]
    Name                forward
    Listen              0.0.0.0
    Port                24224
    tls                 on
    tls.verify          on
    tls.verify_hostname on
    tls.debug           1
    tls.ca_file         ca.crt
    tls.crt_file        centos8-2.crt
    tls.key_file        centos8-2.key
    self_hostname       ${HOSTNAME}

If I change the cert and key to centos8-1.crt and centos8-1.key in centos8-2's config (the INPUT side), the connection cannot be set up with error as below, this makes sense as the CN and the hostname does not match, which is as expected:
centos8-1:

centos8-2:

However, if I change the cert and key on centos8-1's config (the OUTPUT side) to centos8-2.crt and centos8-2.key, the connection can still be set up with logs sending to centos8-2, it seems the tls.verify_hostname on OUTPUT forward does not work.

@cosmo0920 I noticed that you've helped added this function in #8934, could you please help check this issue?

To Reproduce

• In [OUTPUT] forward config, using a cert with different CN than the current server

Expected behavior
OUTPUT tls.verify_hostname should also check the CN and hostname.

Your Environment

• Version used: 3.1.4
• Configuration:
• Environment name and version (e.g. Kubernetes? What version?): virtual machine
• Server type and version: centos8
• Operating System and version: 4.18.0-514.el8.x86_64
• Filters and plugins: [INPUT] and [OUTPUT] forward

[JSchy65]

Maybe related to:
#8072
https://github.com/fluent/fluent-bit/security/advisories/GHSA-rvxv-vp29-qf2h
#8934

[cosmo0920]

Yes. tls.verify_hostname is quite tight condition for some of the server certificates.
This is because some of the server certificates do not have the actual hostnames for their servers.
We use X509_VERIFY_PARAM_set1_host function to verify server's hostname within the used certificate which is described as: https://wiki.openssl.org/index.php/Hostname_validation

And, input and output slightly differ the verification mechanism for certificates.
Both of behavior is just shutdown the TLS connections but how to disconnect the connections are different.

[duj4]

Thanks @cosmo0920 for the reply, per my understanding this behavior (output tls.verify_hostname's not working) is as expected, or do I miss something in the config to make it work correctly?

[duj4]

Thanks @JSchy65 , that's why I opened this issue after checking that change, but seems not working per my understanding.

[mabrarov]

Hi @duj4,

In my case (Fluent Bit 3.1.4) tls.verify_hostname configuration option works well when is turned on (i.e. Fluent Bit fails to send data to the host providing wrong certificate) in Forward output plugin without HA configuration, e.g.:

    [OUTPUT]
        Name            forward
        Match_Regex     ^(kube|host)\..*
        Host            host.minikube.internal
        Port            24224
        Shared_Key      XXX
        Require_ack_response True
        tls         on
        tls.verify  on
        tls.verify_hostname on
        tls.ca_file /fluent-bit/extra/ca.crt
        Retry_Limit     20

But when tls.verify_hostname is turned on and used with HA configuration of Forward output plugin, then Fluent Bit 3.1.4 doesn't work correctly (as expected), i.e. Fluent Bit 3.1.4 successfully sends data to the server providing wrong certificate.

This case seems to be serious security regression from what we have with Fluent Bit 3.0.7 (where tls.verify_hostname configuration option is not available and Fluent But 3.0.7 just always validates CN / SAN of certificate to match hostname).

Here is part of my Fluent Bit 3.1.4 configuration for HA:

    [OUTPUT]
        Name            forward
        Match_Regex     ^(kube|host)\..*
        Upstream        /fluent-bit/extra/upstream.conf
        Retry_Limit     20

and (/fluent-bit/extra/upstream.conf file):

    [UPSTREAM]
        name        forward-balancing
    [NODE]
        name        node-1
        Host        host.minikube.internal
        Port        24224
        Shared_Key  XXX
        Require_ack_response True
        tls         on
        tls.verify  on
        tls.verify_hostname on
        tls.ca_file /fluent-bit/extra/ca.crt
    [NODE]
        name        node-2
        Host        host.minikube.internal
        Port        24225
        Shared_Key  XXX
        Require_ack_response True
        tls         on
        tls.verify  on
        tls.verify_hostname on
        tls.ca_file /fluent-bit/extra/ca.crt

I also tried to put (move, copy) tls.verify_hostname on into [OUTPUT] section from [NODE] section, but it didn't change behavior of Fluent Bit 3.1.4.

So far, my team decided to stay on 3.0.7 version, because TLS is less meaningful if client (Fluent Bit) doesn't validate CN / SAN of server certificate to match expected hostname.

[cosmo0920]

Thanks for the report of verify_hostname settings leaks for HA.
I created a PR to plug this case: #9180

However, this is just in an early stage. Still not handled for memory managements.

[duj4]

@mabrarov thanks for jumping into this issue.

In you first output config, it seems there is no tls.crt_file and tls.crt_key part but only the tls.ca_file, could you please try that as well?

[mabrarov]

Hi @duj4,

Unfortunately, my team doesn't use mTLS b/w Fluent Bit (Forward output plugin) and Fluentd and I do not have example / test of configuration with tls.crt_file and tls.crt_key options in Fluent Bit configuration. It will take some time for me to build and test (Fluent Bit 3.0.7 vs 3.1.4) that configuration, so please do not expect it to happen this week.

Sorry, I realize that the case from my previous message doesn't match your case for 100%, but it is still about tls.verify_hostname and Forward output plugin.

[duj4]

@mabrarov no worry at all and I am not using Fluentd in my case, both output and input are from FluentBit

[cosmo0920]

Thanks @cosmo0920 for the reply, per my understanding this behavior (output tls.verify_hostname's not working) is as expected, or do I miss something in the config to make it work correctly?

If the CN of server is not matched for the provided certificates, the TLS handshake will be failed.
In short, the handshake is expected not to be succeed for wrong certificates and sending logs or other types of payloads will not send via TLS with wrong certificates.
So, if your certificates are wrong for CN of server hostname, the connections will be failed and it's expected behavior.

[duj4]

Thanks @cosmo0920 for the reply, per my understanding this behavior (output tls.verify_hostname's not working) is as expected, or do I miss something in the config to make it work correctly?

If the CN of server is not matched for the provided certificates, the TLS handshake will be failed. In short, the handshake is expected not to be succeed for wrong certificates and sending logs or other types of payloads will not send via TLS with wrong certificates. So, if your certificates are wrong for CN of server hostname, the connections will be failed and it's expected behavior.

yes, this is for sure, but my issue is when I set the mismatched cert in OUTPUT as below, the connection can still be set up:

[OUTPUT]
    Name                forward
    Match               *
    Host                centos8-2
    tls                 on
    tls.verify          on
    tls.verify_hostname on
    tls.debug           1
    tls.ca_file         ca.crt
    tls.crt_file        centos8-2.crt
    tls.key_file        centos8-2.key
    self_hostname       ${HOSTNAME}

The config above is on centos8-1, thus the tls.crt_file and tls.crt_key should match the server that it is running on, isn't it?

[mabrarov]

@duj4,

What about Subject Alternative Name (SAN) of the server certificate in your case? Does it also not match value specified in Host parameter (centos8-2, because tls.verify_hostname is about verification of server certificate, not certificate configured in tls.crt_file parameter)?

[cosmo0920]

Thanks @cosmo0920 for the reply, per my understanding this behavior (output tls.verify_hostname's not working) is as expected, or do I miss something in the config to make it work correctly?

If the CN of server is not matched for the provided certificates, the TLS handshake will be failed. In short, the handshake is expected not to be succeed for wrong certificates and sending logs or other types of payloads will not send via TLS with wrong certificates. So, if your certificates are wrong for CN of server hostname, the connections will be failed and it's expected behavior.

yes, this is for sure, but my issue is when I set the mismatched cert in OUTPUT as below, the connection can still be set up:

[OUTPUT]
    Name                forward
    Match               *
    Host                centos8-2
    tls                 on
    tls.verify          on
    tls.verify_hostname on
    tls.debug           1
    tls.ca_file         ca.crt
    tls.crt_file        centos8-2.crt
    tls.key_file        centos8-2.key
    self_hostname       ${HOSTNAME}

The config above is on centos8-1, thus the tls.crt_file and tls.crt_key should match the server that it is running on, isn't it?

It depends on your SAN value in your certificates.
This can be verified with:

$ openssl  x509 -noout -ext subjectAltName -in /path/to/your_certificates.pem

Then, if there is a SAN extension in your certificates:

X509v3 Subject Alternative Name: 
    DNS:a.super.cool.dns.address

[duj4]

@cosmo0920,

Here is the output of the openssl:

[duj4]

@duj4,

What about Subject Alternative Name (SAN) of the server certificate in your case? Does it also not match value specified in Host parameter (centos8-2, because tls.verify_hostname is about verification of server certificate, not certificate configured in tls.crt_file parameter)?

hi @mabrarov,

Do you mean the tls.verify_hostname is used to verify the destination (the value in Host of [OUPTUT] FORWARD, yes, centos8-2) instead of the server that fluentbit is running on (in my case, it is centos8-1)?

If this is the case, the initial config as below (on centos8-1) should not work (as centos8-1.crt cannot verify the destination centos8-2) but it does work:

[OUTPUT]
    Name                forward
    Match               *
    Host                centos8-2
    tls                 on
    tls.verify          on
    tls.verify_hostname on
    tls.debug           1
    tls.ca_file         ca.crt
    tls.crt_file        centos8-1.crt
    tls.key_file        centos8-1.key
    self_hostname       ${HOSTNAME}

[cosmo0920]

For out_forward, it could be verified for the destination of CN:
https://github.com/fluent/fluent-bit/blob/master/src/tls/flb_tls.c#L525-L556
This is because vhost is came from the destination. flb_upstream_create will be created at:
Simple: https://github.com/fluent/fluent-bit/blob/master/plugins/out_forward/forward.c#L1069
For HA: https://github.com/fluent/fluent-bit/blob/master/src/flb_upstream_node.c#L171

So, the verification of hostname should be used for the destinations on out_forward plugin.

[mabrarov]

Hi @duj4,

tls.verify_hostname is about verification of certificate provided by destination (which is defined by Host parameter).

This part of your configuration:

[OUTPUT]
    ...
    tls.crt_file        centos8-1.crt
    tls.key_file        centos8-1.key

has no relation with verification (performed by Fluent Bit Forward output plugin at centos8-1 host) of certificate provided by destination (located at centos8-2). These parameters are used for mTLS when both client (which is Fluent Bit Forward output plugin running at centos8-1 host) and server (destination, centos8-2) provide certificate.

In case of configuration which you provided only following options impact verification (performed by Fluent Bit Forward output plugin running at centos8-1 host) of certificate provided by destination (of certificate provided by Fluent Bit Forward input plugin running at centos8-2 host):

[OUTPUT]
    ...
    Host                centos8-2
    tls                 on
    tls.verify          on
    tls.verify_hostname on
    ...
    tls.ca_file         ca.crt

[duj4]

hi @mabrarov ,
Thanks for the details, now I am getting more clear on the procedure, but just one more question if you do not mind:

These parameters are used for mTLS when both client (which is Fluent Bit Forward output plugin running at centos8-1 host) and server (destination, centos8-2) provide certificate.

How does server side (centos8-2) verify the client(centos8-1) as there is no such parameter HOST in [INPUT] section?

[duj4]

For out_forward, it could be verified for the destination of CN: https://github.com/fluent/fluent-bit/blob/master/src/tls/flb_tls.c#L525-L556 This is because vhost is came from the destination. flb_upstream_create will be created at: Simple: https://github.com/fluent/fluent-bit/blob/master/plugins/out_forward/forward.c#L1069 For HA: https://github.com/fluent/fluent-bit/blob/master/src/flb_upstream_node.c#L171

So, the verification of hostname should be used for the destinations on out_forward plugin.

Thanks @cosmo0920 for the code lines.

[mabrarov]

Hi @duj4,

I'm not good in mTLS. As far as I know mTLS and as far as I understand Fluent Bit code, the server (centos8-2), when performing verification of client (centos8-1) certificate (in some cases server can skip that verification ignoring certificate provided by client, not sure if it is the case for Fluent Bit Forward input plugin), checks:

1. Validity dates of certificate.
2. Key usage extension of certificate.
3. Extended key usage.
4. Issuer (CA which issued / signed client certificate should be to among CAs which are known to / trusted by server).

Options 1-3 from this list are defined by following part of centos8-1 configuration:

[OUTPUT]
    ...
    tls                 on
    ...
    tls.crt_file        centos8-1.crt
    tls.key_file        centos8-1.key

The set of CAs (CA certificates) which are trusted by server (centos8-2) is defined by following part of centos8-2 configuration (I guess so, I never tried it before):

[INPUT]
    ...
    tls.ca_file         ca.crt

[duj4]

@mabrarov Thanks for the reply.

It seems this mTLS is based on the ca.crt here, as long as the source (centos8-1) is signed by the same CA as destination (centos8-2), it will be trusted.

Please correct me if there is any misunderstanding on how [INPUT] verifies the [OUTPUT] @cosmo0920

[cosmo0920]

Your understanding is correct. Thanks for filing your issue. :)

[duj4]

thanks @cosmo0920 and @mabrarov for your patience, really gained a lot by this one.