[WIP] JUCX: client id support

[petro-rudenko]

What

Follow up on #6859 for JUCX Spark tests

@abellina you can build ucx from this PR and use JUCX from SNAPSHOT of version 1.12.0-client-id-SNAPSHOT

[abellina]

does this mean that client id is not supported for RDMACM?

[petro-rudenko]

No, this ment to set client_id only for ep creation by sockaddr, neither by worker_addr nor by conn_request

[abellina]

@petro-rudenko api makes sense so far, and I can see the clientId populated in the UcpConnectionRequest.

In order to reject a request, do we need to call something? Or just do nothing in the callback (poc patch: abellina/spark-rapids@f156f7b for reference)

I am not able to fully test this right now since I am getting an exception when creating the endpoint:

2021-08-03 07:10:42,159 INFO  UCX:57 - Got UcpListener request from /192.168.50.80:48544 for executorId 1
2021-08-03 07:10:42,167 ERROR UCX:94 - Exception caught in UCX progress thread. Continuing.
org.openucx.jucx.UcxException: Invalid parameter
  at org.openucx.jucx.ucp.UcpEndpoint.createEndpointNative(Native Method)
  at org.openucx.jucx.ucp.UcpEndpoint.<init>(UcpEndpoint.java:42)
  at org.openucx.jucx.ucp.UcpWorker.newEndpoint(UcpWorker.java:52)
  at com.nvidia.spark.rapids.shuffle.ucx.UCX$UcpEndpointManager.onConnectionRequest(UCX.scala:833)
  at org.openucx.jucx.ucp.UcpWorker.progressWorkerNative(Native Method)
  at org.openucx.jucx.ucp.UcpWorker.progress(UcpWorker.java:135)
  at com.nvidia.spark.rapids.shuffle.ucx.UCX.$anonfun$init$5(UCX.scala:187)
  at com.nvidia.spark.rapids.shuffle.ucx.UCX.$anonfun$init$5$adapted(UCX.scala:178)
  at com.nvidia.spark.rapids.Arm.withResource(Arm.scala:28)
  at com.nvidia.spark.rapids.Arm.withResource$(Arm.scala:26)
  at com.nvidia.spark.rapids.shuffle.ucx.UCX.withResource(UCX.scala:69)
  at com.nvidia.spark.rapids.shuffle.ucx.UCX.$anonfun$init$2(UCX.scala:178)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at com.nvidia.spark.rapids.GpuDeviceManager$$anon$1.$anonfun$newThread$1(GpuDeviceManager.scala:340)
  at java.lang.Thread.run(Thread.java:748)

And the UCX log says:

cp_ep.c:778  UCX  ERROR client id is supported only for sockaddr connection establishment

Which seems related to the comment above. I did not specify aUCX_SOCKADDR_TLS_PRIORITY setting, I simply have: UCX_TLS=cuda_copy,cuda_ipc,tcp. Is that expected?

Note I also tried with: UCX_SOCKADDR_TLS_PRIORITY=sockcm,tcp, and still get the same error.

[abellina]

Just updating here. We discussed offline and the issue I had was that I specified the client id both on new endpoints to a remote listener, and to endpoints created as a response of a connection request. If we don't set the client id on the connection request, this error goes away.

I know the API is a bit in flux, there was another issue with what looked like the client id sometimes not being set (i.e. a random high number instead of what I was setting it to). @petro-rudenko any idea on when the new API may be up?

[petro-rudenko]

Closing in favor of #7523