Skip to content

Commit

Permalink
Fix doc, always check ray runtime_env
Browse files Browse the repository at this point in the history
Signed-off-by: Vasily Litvinov <fam1ly.n4me@yandex.ru>
  • Loading branch information
vnlitvinov committed Jul 7, 2022
1 parent 49029e2 commit 2a8b4f7
Show file tree
Hide file tree
Showing 2 changed files with 13 additions and 12 deletions.
7 changes: 3 additions & 4 deletions docs/getting_started/troubleshooting.rst
Original file line number Diff line number Diff line change
Expand Up @@ -291,10 +291,6 @@ or
df = pd.DataFrame([0, 1, 2, 3])
print(df)
.. _issue: https://github.com/modin-project/modin/issues
.. _Slack: https://modin.org/slack.html


Spurious error "cannot import partially initialised pandas module" on custom Ray cluster
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

Expand Down Expand Up @@ -329,3 +325,6 @@ Modin sets up automatically on the Ray clusters it spawns, but it might be missi

So if you're seeing the issue like shown above, please make sure you set this environment variable on all
worker nodes of your cluster before actually spawning the workers.

.. _issue: https://github.com/modin-project/modin/issues
.. _Slack: https://modin.org/slack.html
18 changes: 10 additions & 8 deletions modin/core/execution/ray/common/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -167,14 +167,16 @@ def initialize_ray(
if not GPU_MANAGERS:
for i in range(GpuCount.get()):
GPU_MANAGERS.append(GPUManager.remote(i))
else: # ray is already initialized, check runtime env config
env_vars = ray.get_runtime_context().runtime_env.get("env_vars", {})
for varname, varvalue in extra_init_kw["runtime_env"]["env_vars"].items():
if str(env_vars.get(varname, "")) != str(varvalue):
ErrorMessage.single_warning(
"When using a pre-initialized Ray cluster, please ensure that the runtime env "
+ f"sets environment variable {varname} to {varvalue}"
)

# Now ray is initialized, check runtime env config - especially useful if we join
# an externally pre-configured cluster
env_vars = ray.get_runtime_context().runtime_env.get("env_vars", {})
for varname, varvalue in extra_init_kw["runtime_env"]["env_vars"].items():
if str(env_vars.get(varname, "")) != str(varvalue):
ErrorMessage.single_warning(
"When using a pre-initialized Ray cluster, please ensure that the runtime env "
+ f"sets environment variable {varname} to {varvalue}"
)

num_cpus = int(ray.cluster_resources()["CPU"])
num_gpus = int(ray.cluster_resources().get("GPU", 0))
Expand Down

0 comments on commit 2a8b4f7

Please sign in to comment.