Store: v0.29.0 - Azure - http_config insecure_skip_verify not taken into account.

[ahurtaud]

Thanos, Prometheus and Golang version used:
v0.29.0

Object Storage Provider:
Azure

What happened:
While upgrading from v0.28.1 to v0.29.0, we had some error about TLS certificate not signed (see logs below).
But we are using the insecure_skip_verify config:

      --objstore.config=
        type: AZURE
        config:
          container: "tooling-argos-amadeus-com"
          endpoint: "privatelink.blob.core.windows.net"
          storage_account: "$(OBJSTORE_ACCESS_KEY)"
          storage_account_key: "$(OBJSTORE_SECRET_KEY)"
          max_retries: 0
          http_config:
            insecure_skip_verify: true

What you expected to happen:
Update from v0.28.1 to v0.29.0 should not break thanos store connectivity to azure blob while using insecure_skip_connectivity.

How to reproduce it (as minimally and precisely as possible):
It seems this http_config flag is not taken into account with the new Azure connectivity?
related to #5707 @phillebaba ?

single revert to v0.28.1 fixed the thanos store component :/ (same config, same flag)

Full logs to relevant components:

Logs

level=info ts=2022-11-07T14:41:49.500916635Z caller=factory.go:52 msg="loading bucket configuration" level=error ts=2022-11-07T14:42:34.959925537Z caller=main.go:135 err="===== INTERNAL ERROR =====\nGet \"https://1aargoststne.privatelink.blob.core.windows.net/system-argos-amadeus-com?restype=container\": x509: certificate is valid for *.blob.core.windows.net, *.dub12prdstr11a.store.core.windows.net, *.blob.storage.azure.net, *.z1.blob.storage.azure.net, *.z2.blob.storage.azure.net, *.z3.blob.storage.azure.net, *.z4.blob.storage.azure.net, *.z5.blob.storage.azure.net, *.z6.blob.storage.azure.net, *.z7.blob.storage.azure.net, *.z8.blob.storage.azure.net, *.z9.blob.storage.azure.net, *.z10.blob.storage.azure.net, *.z11.blob.storage.azure.net, *.z12.blob.storage.azure.net, *.z13.blob.storage.azure.net, *.z14.blob.storage.azure.net, *.z15.blob.storage.azure.net, *.z16.blob.storage.azure.net, *.z17.blob.storage.azure.net, *.z18.blob.storage.azure.net, *.z19.blob.storage.azure.net, *.z20.blob.storage.azure.net, *.z21.blob.storage.azure.net, *.z22.blob.storage.azure.net, *.z23.blob.storage.azure.net, *.z24.blob.storage.azure.net, *.z25.blob.storage.azure.net, *.z26.blob.storage.azure.net, *.z27.blob.storage.azure.net, *.z28.blob.storage.azure.net, *.z29.blob.storage.azure.net, *.z30.blob.storage.azure.net, *.z31.blob.storage.azure.net, *.z32.blob.storage.azure.net, *.z33.blob.storage.azure.net, *.z34.blob.storage.azure.net, *.z35.blob.storage.azure.net, *.z36.blob.storage.azure.net, *.z37.blob.storage.azure.net, *.z38.blob.storage.azure.net, *.z39.blob.storage.azure.net, *.z40.blob.storage.azure.net, *.z41.blob.storage.azure.net, *.z42.blob.storage.azure.net, *.z43.blob.storage.azure.net, *.z44.blob.storage.azure.net, *.z45.blob.storage.azure.net, *.z46.blob.storage.azure.net, *.z47.blob.storage.azure.net, *.z48.blob.storage.azure.net, *.z49.blob.storage.azure.net, *.z50.blob.storage.azure.net, not 1aargoststne.privatelink.blob.core.windows.net\nAzure API return unexpected error: *azblob.InternalError\n\ngh.neting.cc/thanos-io/objstore/providers/azure.NewBucketWithConfig\n\t/go/pkg/mod/github.com/thanos-io/objstore@v0.0.0-20221006135717-79dcec7fe604/providers/azure/azure.go:170\ngh.neting.cc/thanos-io/objstore/providers/azure.NewBucket\n\t/go/pkg/mod/github.com/thanos-io/objstore@v0.0.0-20221006135717-79dcec7fe604/providers/azure/azure.go:150\ngh.neting.cc/thanos-io/objstore/client.NewBucket\n\t/go/pkg/mod/github.com/thanos-io/objstore@v0.0.0-20221006135717-79dcec7fe604/client/factory.go:70\nmain.runStore\n\t/app/cmd/thanos/store.go:254\nmain.registerStore.func1\n\t/app/cmd/thanos/store.go:198\nmain.main\n\t/app/cmd/thanos/main.go:133\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594\ncreate AZURE client\ngh.neting.cc/thanos-io/objstore/client.NewBucket\n\t/go/pkg/mod/github.com/thanos-io/objstore@v0.0.0-20221006135717-79dcec7fe604/client/factory.go:87\nmain.runStore\n\t/app/cmd/thanos/store.go:254\nmain.registerStore.func1\n\t/app/cmd/thanos/store.go:198\nmain.main\n\t/app/cmd/thanos/main.go:133\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594\ncreate bucket client\nmain.runStore\n\t/app/cmd/thanos/store.go:256\nmain.registerStore.func1\n\t/app/cmd/thanos/store.go:198\nmain.main\n\t/app/cmd/thanos/main.go:133\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594\npreparing store command failed\nmain.main\n\t/app/cmd/thanos/main.go:135\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"

[phillebaba]

Most likely correct that the SDK upgrade has caused this. I will see if I can reproduce this. Could we move this issue to the objstore repository as that is where the relevant code lives?

[ahurtaud]

I dont have permissions to do so, @GiedriusS can you transfer this issue to https://github.com/thanos-io/objstore please?
sorry I just discovered the objstore repo has been created :/

[ahurtaud]

@phillebaba I would appreciate if you can update the go module on thanos main so we can close this issue (and make sure the fix is part of any patch or next release).
or ping me, if you want me to PR this.