Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: virtual_partition.wait failed if there are duplicate refs #5045

Closed
3 tasks done
anmyachev opened this issue Sep 27, 2022 · 0 comments · Fixed by #5058
Closed
3 tasks done

BUG: virtual_partition.wait failed if there are duplicate refs #5045

anmyachev opened this issue Sep 27, 2022 · 0 comments · Fixed by #5058
Assignees
Labels
bug 🦗 Something isn't working P2 Minor bugs or low-priority feature requests Ray ⚡ Issues related to the Ray engine

Comments

@anmyachev
Copy link
Collaborator

Modin version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest released version of Modin.

  • I have confirmed this bug exists on the main branch of Modin. (In order to do this you can follow this guide.)

Reproducible Example

import numpy as np
import modin.pandas as pd

frame_c = pd.DataFrame(np.zeros((100, 20), dtype=np.float32, order="C"))
frame_c = [frame_c] * 20
df = pd.concat(frame_c)
partitions = df._query_compiler._modin_frame._partitions.flatten()
if hasattr(partitions[0], "wait"):
    all(map(lambda partition: partition.wait(), partitions))
print(df)

Issue Description

ray.wait doesn't work with duplicated refs, although it is common for pandas operations to use the same objects.

Expected Behavior

The producer must work without errors.

Error Logs

Traceback (most recent call last):
  File "test_wait.py", line 9, in <module>
    all(map(lambda partition: partition.wait(), partitions))
  File "test_wait.py", line 9, in <lambda>
    all(map(lambda partition: partition.wait(), partitions))
  File "C:\projects\modin\modin\core\execution\ray\implementations\pandas_on_ray\partitioning\virtual_partition.py", line 472, in wait
    ray.wait(futures, num_returns=len(futures))
  File "C:\Users\amyachev\Miniconda3\envs\modin\lib\site-packages\ray\_private\client_mode_hook.py", line 105, 
in wrapper
    return func(*args, **kwargs)
  File "C:\Users\amyachev\Miniconda3\envs\modin\lib\site-packages\ray\worker.py", line 1982, in wait
    raise ValueError("Wait requires a list of unique object refs.")
ValueError: Wait requires a list of unique object refs.

Installed Versions

INSTALLED VERSIONS

commit : 08a41d678490314db6f2ee95fe434fbaef90f93b
python : 3.8.13.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19044
machine : AMD64
processor : Intel64 Family 6 Model 140 Stepping 1, GenuineIntel
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : English_United States.1252

Modin dependencies

modin : 0.15.0+159.g08a41d67
ray : 1.12.1
dask : 2022.01.1
distributed : 2022.01.1
hdk : None

pandas dependencies

pandas : 1.4.4
numpy : 1.23.0
pytz : 2022.1
dateutil : 2.8.2
setuptools : 59.8.0
pip : 22.1.2
Cython : None
pytest : 7.1.2
hypothesis : None
sphinx : 5.0.2
blosc : None
feather : 0.4.1
xlsxwriter : None
lxml.etree : 4.9.0
html5lib : None
pymysql : None
psycopg2 : 2.9.3
jinja2 : 3.1.2
IPython : 8.4.0
pandas_datareader: None
bs4 : 4.11.1
bottleneck : None
brotli :
fastparquet : None
fsspec : 2022.5.0
gcsfs : None
markupsafe : 2.1.1
matplotlib : 3.2.2
numba : None
numexpr : 2.7.3
odfpy : None
openpyxl : 3.0.9
pandas_gbq : 0.17.6
pyarrow : 8.0.0
pyreadstat : None
pyxlsb : None
s3fs : 2022.5.0
scipy : 1.8.1
snappy : None
sqlalchemy : 1.4.39
tables : 3.7.0
tabulate : None
xarray : 2022.3.0
xlrd : 2.0.1
xlwt : None
zstandard : None

@anmyachev anmyachev added bug 🦗 Something isn't working Triage 🩹 Issues that need triage Ray ⚡ Issues related to the Ray engine labels Sep 27, 2022
@pyrito pyrito added P2 Minor bugs or low-priority feature requests and removed Triage 🩹 Issues that need triage labels Sep 27, 2022
@noloerino noloerino self-assigned this Sep 28, 2022
noloerino added a commit to noloerino/modin that referenced this issue Sep 28, 2022
… object refs

Signed-off-by: Jonathan Shi <jhshi@ponder.io>
YarShev pushed a commit that referenced this issue Oct 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug 🦗 Something isn't working P2 Minor bugs or low-priority feature requests Ray ⚡ Issues related to the Ray engine
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants