Receive: stop relying on grpc server config to set grpc client secure/skipVerify

[guillaumelecerf]

• I added CHANGELOG entry for this change.
• Change is not relevant to the end user.

Changes

I'm using a thanos receive HA cluster on ECS Fargate, with 3 endpoints in the hashring, with one dedicated listener + grpc target group per endpoint.

The TLS termination is done at the listener level (so out of thanos). Therefore the thanos receive containers are configured with no TLS.

When testing, I ended up with error reading server preface: http2: frame too large errors.

After debugging, I found that the receive grpc client is configured in secure mode only if the receive grpc server has TLS enabled.
This breaks the inter-endpoint communication between our hashring members.

To fix this problem, I introduce 2 new flags --remote-write.client-tls-secure and --remote-write.client-tls-skip-verify to mimic what is done in the grpc client configuration of thanos query..

Verification

Everything is now working on our cluster.

[guillaumelecerf]

Hi,
Can someone please have a look?

[fpetkovski]

I did not fully understand the TLS setup, at which point in Thanos do you have TLS enabled?

[guillaumelecerf]

I did not fully understand the TLS setup, at which point in Thanos do you have TLS enabled?

@fpetkovski : thanks for having a look.

In my setup, thanos is never configured with TLS, as the TLS termination is done externally at the listener level in AWS.

The problem here is that the code assumes that the gRPC client (used when talking with the other members of the hashring) is using TLS only if the server is configured with TLS.

For my use case, these 2 notions must be decoupled, as it is done in query.

[fpetkovski]

Ok, this makes sense, and is also consistent with the query configuration.