Stop ray to write temp files during simulation

[charlesbvll]

Discussed in https://github.com/adap/flower/discussions/1394

^{Originally posted by davideNasi August 29, 2022}
I'm using torch implementation of TabNet (https://github.com/dreamquark-ai/tabnet) and when I run simulations each client write a 1.6 GB file every epoch in C:\Users<user>\AppData\Local\Temp\ray\session_\ray_spilled_objects.

Am I doing something wrong or is it possible to prevent this?

Current workaround

Downgrade ray to version 1.11.1 : pip install ray==1.11.1

Possible solutions

• Find a way to disable object spilling in the current version of ray
• Escalate the issue to the ray-project
• Change the ray version to 1.11.1 in the pyproject.toml

[charlesbvll]

Update

From, https://discuss.ray.io/t/object-store-spilling-terabytes-of-data/8699, it seems that upcoming Ray version 2.3 should resolve the issue.
After some thorough testing with Ray 3.0 nightly, it seems to resolve the issue.

The Ray PR that solved the problem is this one: https://github.com/ray-project/ray/pull/31488/files

If you don't want to wait for Ray to officially release their next version, you can try out the nightly by following this guide:
https://docs.ray.io/en/latest/ray-overview/installation.html#daily-releases-nightlies