Proxy cannot be used when deployed to the Linux

[lileixer]

Hi
I configured the proxy and it was available locally, but when deployed on a Linux server the program threw an exception
The error message is：(StatusCode="Unavailable", Detail="failed to connect to all addresses", DebugException="Grpc.Core.Internal.CoreErrorDetailException: {"created":"@1634714686.033195058","description":"Failed to pick subchannel","file":"/var/local/git/grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":5397,"referenced_errors":[{"created":"@1634714686.033187157","description":"failed to connect to all addresses","file":"/var/local/git/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":398,"grpc_status":14}]}"). Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="failed to connect to all addresses", DebugException="Grpc.Core.Internal.CoreErrorDetailException: {"created":"@1634714686.033195058","description":"Failed to pick subchannel","file":"/var/local/git/grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":5397,"referenced_errors":[{"created":"@1634714686.033187157","description":"failed to connect to all addresses","file":"/var/local/git/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":398,"grpc_status":14}]}")

[AnashOommen]

@lileixer I use what the grpc library provides, so let me ask someone from the team to chime in. What proxy string are you using? Could you share it?

[lileixer]

Thanks a lot. Of course I would.But this is a private proxy, I hope I can send it to you separately. My email is leilichn005@gmail.com, please send me an email and I will reply to you.

[jtattermusch]

Some general advice that might help:

• You might get some extra insights by setting GRPC_VERBOSITY=debug env variable and then inspecting the logs, as described here:
  https://github.com/grpc/grpc/blob/master/TROUBLESHOOTING.md#grpc_verbosity. You can also use GRPC_TRACE=tcp to see the actual bytes being sent by gRPC (it might help to debug the problem).
• are you sure that the proxy you are using supports HTTP/2? This is a requirements for being able to proxy grpc traffic (as gRPC is based on HTTP/2 protocol).
• are you using secure or insecure connection? Setting up insecure proxy is easier, so that's where I would start with experiments.
• seeing at least the format of the URL you are using for http_proxy would help. Can you share what software is the proxy?

relevant proxy configuration code:
https://github.com/grpc/grpc/blob/c8db1091b8c02f5cf7ac1cb2f30363141ce28023/src/core/ext/filters/client_channel/http_proxy.cc#L71

[lileixer]

@jtattermusch
I use the proxy software is clash, https://github.com/Dreamacro/clash/releases/tag/premium
I tried to configure the debug environment variable, but I could not find the reason from the log.
I can share the proxy string with you separately. If necessary, you can send me an email. The email address is leilichn005@gmail.com

[jtattermusch]

From the documentation it's not obvious to me whether the proxy you use supports HTTP 2.0
https://github.com/Dreamacro/clash/blob/master/README.md

Can you share the obfuscated proxy URL here (e.g. replace IP address with random number or replace hostname with random characters, just keep the structure of the URL)?

[lileixer]

@jtattermusch
Proxy: http://xxx.xxx.xxx.xxx:8080
It's definitely supported, and the GoogleAds sample works fine on a Windows development environment

[jtattermusch]

@jtattermusch Proxy: http://xxx.xxx.xxx.xxx:8080 It's definitely supported, and the GoogleAds sample works fine on a Windows development environment

Ok, so I'm I understanding correctly that the exact same setup works fine on Windows and it doesn't work on Linux (Linux vs Windows being the only difference here?)

[lileixer]

@jtattermusch
That's true.

[jtattermusch]

@jtattermusch That's true.

Any interesting differences in the logs when GRPC_VERBOSITY=debug and GRPC_TRACE=tcp are set? On linux there should be some logs that suggest why connecting has failed (bad security handshake? ip unreachable from linux?) or some difference in the bytes that gets sent once the proxy is established.

[lileixer]

@jtattermusch
I'll show you the log after I reproduce the problem

[lileixer]

@jtattermusch

After doing the configuration, the log is as follows
windows.txt
linux.txt

[jtattermusch]

The lack of useful linux logs seems surprising. Can you try with GRPC_TRACE=handshaker and also GRPC_DNS_RESOLVER=native (since the windows log mentions that it's using native DNS resolver, which isn't the default btw).

Which Grpc.Core version are you using?

[jtattermusch]

Ok, I think I know what's wrong. Don't use Environment.SetEnvironmentVariable() on linux.
It doesn't actually set the env values the internal native library sees (see grpc/grpc#14963 for details).
To set env variables on linux, set them on command line or in a shell script.

this might be the reason why no useful logs are printed on linux and why http_proxy on linux appears to not work (the real problem is that the C-core library doesn't see those env variables being set).

Also see grpc/grpc#19706

[lileixer]

Thanks, let me try

[lileixer]

@jtattermusch
Thank you very much!
When I run "export http_proxy=http://myproxy.com:7890" command on Linux to set the environment variables, the program runs smoothly and the problem is solved.

[jtattermusch]

Let's close this issue then?

[lileixer]

OK

[lileixer]

@jtattermusch

Today I encountered a new problem. There are two places in my project that need to use a proxy, one is GoogleSDK and the other is for other platforms. The two places need to use different proxy addresses, and the environment variable configuration cannot be isolated. How can I Does the operation make it read the project configuration? (Project deployment to Linux)

[jtattermusch]

If you have access to the creation of gRPC channels and you can add extra configuration,
you can set the proxy on per-channel basis via a channel argument (or "ChannelOption" in gRPC C#).

During creating of the channel, you can pass new ChannelOption("grpc.http_proxy","YOUR_PROXY_URL")
https://github.com/grpc/grpc/blob/09658682c93805d5114a0d638388349c4276f9a3/include/grpc/impl/codegen/grpc_types.h#L418

If you don't have access to the channel creation (not sure if the ads client library provides that), you're out of luck, since the only other option is setting the enviroment variable.

[lileixer]

Looking at the source code, the CreateChannel packaged by the SDK still implements the proxy by setting environment variables. Does this mean that the existing SDK can't do what I want to do

[lileixer]

I'm trying to pull a branch and change the implementation

[AnashOommen]

Oh... That's a very useful information...I can provide a fix in the library to use ChannelOptions instead of environment variables.

…

On Thu, Oct 28, 2021, 3:42 AM Jan Tattermusch ***@***.***> wrote: If you have access to the creation of gRPC channels and you can add extra configuration, you can set the proxy on per-channel basis via a channel argument (or "ChannelOption" in gRPC C#). During creating of the channel, you can pass new ChannelOption("grpc.http_proxy","YOUR_PROXY_URL") https://github.com/grpc/grpc/blob/09658682c93805d5114a0d638388349c4276f9a3/include/grpc/impl/codegen/grpc_types.h#L418 If you don't have access to the channel creation (not sure if the ads client library provides that), you're out of luck, since the only other option is setting the enviroment variable. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#320 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCOJCEJSWKDZN7X22QGNZTUJEEFZANCNFSM5GK54L5Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[lileixer]

Looking forward to your update