where to set HTTP_PROXY in python app?

[swuecho]

Thanks for this library. really make grpc more easy to use.

I am trying the example with my python sever. have no idea what is the http_proxy for. If I let the grpc_dump connect to the port my python server listening to. The request are show, but no any response.

my python server is simliar to:

https://github.com/grpc/grpc/blob/master/examples/python/helloworld/greeter_server_with_reflection.py

clearly I miss something. I am trying to find the demo app you are using. Could you put the echo app code as an example? Thanks.

[bradleyjkemp]

Hi, HTTP_PROXY is an environment variable that the python gRPC should automatically read and use (see https://github.com/grpc/grpc/blob/master/doc/environment_variables.md).

So you should be able to do something in a terminal like:

# Terminal 1
python greeter_server.py

# Terminal 2
export HTTP_PROXY=localhost:<grpc_dump_port>
python greeter_client.py

Could you see if this works?

And thanks for the question! I'll update the docs to make it clear that HTTP_PROXY is an environment variable

[bradleyjkemp]

Ah, another thing that might be causing an issue: proxies are normally ignored for localhost connections.
Instead of the client pointing at localhost:50051, you could use something like http://readme.localtest.me (e.g. point it at localhost.me:50051) or edit your hosts file to add an entry like my-local-server.com 127.0.0.1

[swuecho]

probably because my client is grpc web? so it can not read the http_proxy?

[bradleyjkemp]

Could you give me the grpc-dump command that you're running?

You should be able to configure your browser to use grpc-dump as a HTTP proxy and then it will be able to log all requests and responses. If you set up both HTTP and HTTPS proxy using these instructions https://software.intel.com/en-us/articles/how-to-set-system-proxy your browser should use the proxy.

[swuecho]

Thanks for the instruction. I start to understand what is going on.

http_proxy=localhost:$DUMP_PORT my_app is making an connnetion to $DUMP_PORT, i.e the proxy.

telling the proxy, my_app, want to talk to the server at $SERVER_PORT. proxy then talk to the server.

the network flow in my situation is like this: web (grpc-web) --80--> envoy --9090-> grpc-server.

if I put run grpc-dump --port=9090, then I see the request are displayed in the grpc-dump output.
(obviously, grpc-dump is doing wonderful job here). Here, the app is treating the grpc-dump as the server.

I did not tell the grpc-web (or envoy more precisely) that it is a proxy.

In order to make it work, have to find some way to config the envoy tell it about the proxy.

then the whole network flow is like this:

web (grpc-web) --80-- envoy---$DUMP_PORT--grpc-dump--9090-grpc-server

If grpc-dump can have an in port and an out port. I could run something like grpc-dump --client-port=9090 --server-port=9091 and change the server port to 9091.
this will be ideal and requires minimal set up. (if I can find a way to set up envoy, that will be best)

[bradleyjkemp]

Yup I think you've got the right idea. grpc-dump is a HTTP CONNECT proxy so clients connect to it and send a request:

CONNECT server:port HTTP/1.1

grpc-dump then makes a connection to that server and all future bytes sent on the connection are proxied (but we eavesdrop the connection to be able to get the gRPC request data)

I think grpc-dump can already do what you're looking for using the --destination flag (this should behave the same as your proposed --server-port). This lets you specify where received connections should be sent to.
So you could try something like:

grpc-dump --port=9090 --destination=localhost:9091

[swuecho]

I tried the destination option and have difficulty to get it working. so tried the destination flag on the hello world: https://github.com/swuecho/hello_grpc

when use grpc_dump --port=$PORT and http_proxy=http://localhost:$PORT python greeter_client.py. works great.

but use destination flag seem have some problem. Is there a test case for this situation?

[bradleyjkemp]

Aha! Thanks for the repo link, I've found a bug with the --destination flag but I've got your example working now. I'll be releasing a v0.1.2 shortly.

Test coverage definitely isn't a strong point at the moment 🙁

[bradleyjkemp]

Ok, so as of v0.1.2 you have two options:

1. Use grpc-proxy --port=<proxy> --destination=localhost:<server> and the client connects to localhost:<proxy> (without any HTTP_PROXY environment variables set)
2. Use grpc-proxy --port=<proxy> and the client connects to localhost:<server> but with the environment variable http_proxy=http://localhost:<proxy> set.

Looking at the gRPC library code, you're right that python is a bit pickier than Go when it comes proxies so you have to use lower case http_proxy and specify http:// in the value

Using grpc-dump v0.1.2 both of these cases are working on my machine with just the port changes in greeter_client.py described.

[swuecho]

Thanks. I verified the option 1 works for grpc-web workflow. This is important for grpc-web users, because grpc-web project usually use envoy proxy. I did not find an option to tell envoy proxy to use http_proxy. As a result, no control in the client side (even it has such an option, I do not want to play with the complex yaml config).