Get rid of redundant function calls on Ray workers

[YarShev]

We have had a hacky solution to prevent #3014, #647, #746. We should try to get rid of it since we use Ray functionality that is not directly supposed to be used.

[vnlitvinov]

Is the issue we're working around really gone?.. If it isn't this is going to hamper the stability a lot (and my impression is the issue in Ray is still there).

[YarShev]

I do not see the issue locally and there is no issue in our CI. Also, there is an issue #3601 where a run of pandas imports on Ray workers seems to be a problem.

[devin-petersohn]

It's most evident on large machines with 100+ CPU cores. I don't trust that the bug is gone in Ray as long as the import thread is separate from the worker thread. I prefer to leave the code as it is.

Is there a problem with using these functionalities in Ray?

[YarShev]

I had run Modin tests on a machine with 112 CPU cores and didn't see any fail. Let's wait @rkooo567's answer in #3601 on the status of the problem.

Is there a problem with using these functionalities in Ray?

I prefer Modin (and probably other users/packages) used high-level API (link) from Ray and other packages and didn't not rely on their internals.

[devin-petersohn]

I am strongly against removing it. I spent probably 100 hours debugging and trying to get to a solution for the import thread race condition, even when engaging with the Ray team. Yes, it's not ideal to use internal APIs, but it is too important and I have spent too much time on it. It's not something that's even easily reproduced, and as long as Ray chooses to import in a separate thread from running actual code, I do not trust that design for Modin workloads.

[YarShev]

Makes sense. Let's follow the discussion in #3601 for now.

[RehanSD]

Hi @YarShev @vnlitvinov @devin-petersohn has this issue been resolved?

[mvashishtha]

@RehanSD I think #4603 fixed this.