wip: socks 5 test

[mario-s]

Closes #110

• new test dependency

• check if environment variables to the Kafka cluster and proxy are set

• if not test will be skipped

• values for cluster and proxy are taken from env variables

• add sample in README

• I've read the contributing section of the project CONTRIBUTING.md.

• Signed CLA

[crepererum]

For #110 to be closed we need some SOCKS5 test running within CI as well. Could you have a look at the CircleCI configs and see if you find a way to get this to work?

Also it would be nice if we can offer some documentation on how to test this locally. There are two docker compose files (one for Kafka and one for redpanda) and I think that would be a good place to integrate some SOCKS5 proxy running in a container.

[crepererum]

I consider that a pretty major drawback. This is the reason why we use some custom macro for the other environment variables. Would be great if you could use a similar approach.

[mario-s]

Sure, I changed that to use a macro. However the drawback of this approach is, that you don't see that the test is skipped.

[crepererum]

Yeah, the macro is not ideal. My hope is that rust-lang/rfcs#3221 will be accepted and we can get the best of both worlds. But until then the skipping won't be reported properly.

[yanganto]

Hi,
this is something you may want to use for this scenario.
https://crates.io/crates/test-with

If you are using CI tool for testing, or under development with source code changing, you do not need cargo clean before testing.

Besides, the test report from cargo will change to the following format for a better understanding.

running 2 tests
test tests::test_ignored ... ignored, because variable not found: NOTHING
test tests::test_works ... ok

test result: ok. 1 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.00s

If there is any issue on the crate, an issue is always welcome. :)

[yanganto]

By the way, there is more than one issue in #[ignore(if)], so it might not be the better solution to solve. It is good to keep ignoring in runtime and ignoring under conditions separately.

The following are related PR and issues in Rust, for you to know more about these if you are interested.

• Add ability to ignore tests at runtime. rust-lang/rust#68007
• Proof of concept: implement test ignore-by-panic rust-lang/rust#96574

[crepererum]

I'm not to worried about the CI and the env variable handling there. Ignoring tests and build time instead of runtime is IMHO a messy DX (developer experience). Rskafka is already a somewhat complicated crate -- especially when you consider the integration tests. Sure, not having a proper "ignore" status for skipped tests is bad DX as well, but I consider this the lesser evil.

So let's wait until Rust implements the required feature. I've created #139 to track this on our side.

[mario-s]

You could assign the #139 to me as well, I would be more than happy to integrate the feature, when Rust is ready.

[mario-s]

I tested it with "Charles" on my local machine, before opening this PR. Unfortunately I was not able to integrate any SOCKS5 Proxy into the docker compose file for the Kafka cluster yet.

[mario-s]

Update: I found a Rust SOCK5 proxy that works. It can be easily installed via Cargo. But if you like a Docker image, I think it is better to create a new one, since the existing images are rather old or contain no hint about the version.

[crepererum]

@mario-s what about https://hub.docker.com/r/serjs/go-socks5-proxy/ ? That seems to be a simple proxy packaged as a docker image.

[mario-s]

@crepererum Thanks for the hint. I already tried the that proxy server in docker. Included in the docker-compose file, as well as a standalone docker container. But a connection to the Kafka cluster could not be established during the test.
I ran the test with this command: TEST_INTEGRATION=1 KAFKA_CONNECT=0.0.0.0:9093 SOCKS_PROXY=localhost:1080 cargo test --features transport-socks5
My next attempt was to pass the name of the container for the KAFKA_CONNECT variable, because I thought the Proxy just doesn't see this IP. But then all other integration test, which use the cluster, did not pass.
Therefore I'm not even sure if there is a way to embed a proxy into the docker-compose.

I also tried out the Kafka-Proxy project, same result, but different cause.

[crepererum]

@mario-s I see, it's a bit of mess with the compose files because they bind the brokers to localhost. I've played around with this is and came up with a solution in #135: It basically adds new broker identities/hosts that are visible to the proxy. Funnily this is way eaiser for CircleCI because there we have true virtual host names for the brokers that are resolvable by the client. Another trick that I've added is that KAFKA_CONNECT now accepts a list of brokers which also allows us to pass in an invalid broker just to test that everything still works if we're doing that. I cannot add you as a reviewer (IIRC this only works for org-members) but it would appreciate if you could have a look and add your review.

[crepererum]

#135 is now merged. Thanks again to @mario-s for the work that was also included the merged PR.