Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

submitit fails on python3.6 #624

Closed
jieru-hu opened this issue May 29, 2020 · 9 comments
Closed

submitit fails on python3.6 #624

jieru-hu opened this issue May 29, 2020 · 9 comments
Labels
bug Something isn't working plugin Plugins realted issues

Comments

@jieru-hu
Copy link
Contributor

It seems like it's related to the fact the hydra project is typed, reason being: 1) same tests worked on https://github.com/fairinternal/hydra-fair-plugins (no typing) 2) error message

(pytest36) jieru-mbp:example jieru$ python my_app.py task=1,2,3 -m
[2020-05-29 10:34:33,074][HYDRA] Sweep output dir : multirun/2020-05-29/10-34-33
[2020-05-29 10:34:33,074][HYDRA]        #0 : task=1
[2020-05-29 10:34:33,075][HYDRA]        #1 : task=2
[2020-05-29 10:34:33,075][HYDRA]        #2 : task=3
Traceback (most recent call last):
  File "my_app.py", line 20, in <module>
    my_app()
  File "/Users/jieru/workspace/hydra-fork/hydra/hydra/main.py", line 37, in decorated_main
    strict=strict,
  File "/Users/jieru/workspace/hydra-fork/hydra/hydra/_internal/utils.py", line 261, in run_hydra
    lambda: hydra.multirun(
  File "/Users/jieru/workspace/hydra-fork/hydra/hydra/_internal/utils.py", line 185, in run_and_report
    func()
  File "/Users/jieru/workspace/hydra-fork/hydra/hydra/_internal/utils.py", line 264, in <lambda>
    overrides=args.overrides,
  File "/Users/jieru/workspace/hydra-fork/hydra/hydra/_internal/hydra.py", line 135, in multirun
    return sweeper.sweep(arguments=task_overrides)
  File "/Users/jieru/workspace/hydra-fork/hydra/hydra/_internal/core_plugins/basic_sweeper.py", line 113, in sweep
    results = self.launcher.launch(batch, initial_job_idx=initial_job_idx)
  File "/Users/jieru/workspace/hydra-fork/hydra/plugins/hydra_submitit_launcher/hydra_plugins/hydra_submitit_launcher/submitit_launcher.py", line 139, in launch
    return [j.results()[0] for j in jobs]
  File "/Users/jieru/workspace/hydra-fork/hydra/plugins/hydra_submitit_launcher/hydra_plugins/hydra_submitit_launcher/submitit_launcher.py", line 139, in <listcomp>
    return [j.results()[0] for j in jobs]
  File "/Users/jieru/opt/anaconda3/envs/pytest36/lib/python3.6/site-packages/submitit/core/core.py", line 286, in results
    outcome, result = self._get_outcome_and_result()
  File "/Users/jieru/opt/anaconda3/envs/pytest36/lib/python3.6/site-packages/submitit/core/core.py", line 366, in _get_outcome_and_result
    f"Job {self.job_id} (task: {self.task_id}) with path {self.paths.result_pickle}\n"
submitit.core.utils.UncompletedJobError: Job 80923 (task: 0) with path /Users/jieru/workspace/hydra-fork/hydra/plugins/hydra_submitit_launcher/example/multirun/2020-05-29/10-34-33/.local/80923_0_result.pkl
has not produced any output (state: FINISHED)
Error stream produced:
----------------------
submitit ERROR (2020-05-29 10:34:34,657) - Could not dump error:
Can't pickle typing.Dict[str, str]: it's not the same object as typing.Dict

because of A temporary saved file already exists.
submitit ERROR (2020-05-29 10:34:34,657) - Submitted job triggered an exception
Traceback (most recent call last):
  File "/Users/jieru/opt/anaconda3/envs/pytest36/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Users/jieru/opt/anaconda3/envs/pytest36/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/jieru/opt/anaconda3/envs/pytest36/lib/python3.6/site-packages/submitit/core/_submit.py", line 11, in <module>
    submitit_main()
  File "/Users/jieru/opt/anaconda3/envs/pytest36/lib/python3.6/site-packages/submitit/core/submission.py", line 65, in submitit_main
    process_job(args.folder)
  File "/Users/jieru/opt/anaconda3/envs/pytest36/lib/python3.6/site-packages/submitit/core/submission.py", line 58, in process_job
    raise error
  File "/Users/jieru/opt/anaconda3/envs/pytest36/lib/python3.6/site-packages/submitit/core/submission.py", line 49, in process_job
    utils.pickle_dump(("success", result), tmppath)
  File "/Users/jieru/opt/anaconda3/envs/pytest36/lib/python3.6/site-packages/submitit/core/utils.py", line 278, in pickle_dump
    pickle.dump(obj, ofile, pickle.HIGHEST_PROTOCOL)
_pickle.PicklingError: Can't pickle typing.Dict[str, str]: it's not the same object as typing.Dict

(pytest36) jieru-mbp:example jieru$ python my_app.py task=1,2,3 -m
[2020-05-29 10:35:45,059][HYDRA] Sweep output dir : multirun/2020-05-29/10-35-45
[2020-05-29 10:35:45,060][HYDRA]        #0 : task=1
[2020-05-29 10:35:45,060][HYDRA]        #1 : task=2
[2020-05-29 10:35:45,060][HYDRA]        #2 : task=3
Traceback (most recent call last):
  File "my_app.py", line 20, in <module>
    my_app()
  File "/Users/jieru/workspace/hydra-fork/hydra/hydra/main.py", line 37, in decorated_main
    strict=strict,
  File "/Users/jieru/workspace/hydra-fork/hydra/hydra/_internal/utils.py", line 261, in run_hydra
    lambda: hydra.multirun(
  File "/Users/jieru/workspace/hydra-fork/hydra/hydra/_internal/utils.py", line 185, in run_and_report
    func()
  File "/Users/jieru/workspace/hydra-fork/hydra/hydra/_internal/utils.py", line 264, in <lambda>
    overrides=args.overrides,
  File "/Users/jieru/workspace/hydra-fork/hydra/hydra/_internal/hydra.py", line 135, in multirun
    return sweeper.sweep(arguments=task_overrides)
  File "/Users/jieru/workspace/hydra-fork/hydra/hydra/_internal/core_plugins/basic_sweeper.py", line 113, in sweep
    results = self.launcher.launch(batch, initial_job_idx=initial_job_idx)
  File "/Users/jieru/workspace/hydra-fork/hydra/plugins/hydra_submitit_launcher/hydra_plugins/hydra_submitit_launcher/submitit_launcher.py", line 139, in launch
    return [j.results()[0] for j in jobs]
  File "/Users/jieru/workspace/hydra-fork/hydra/plugins/hydra_submitit_launcher/hydra_plugins/hydra_submitit_launcher/submitit_launcher.py", line 139, in <listcomp>
    return [j.results()[0] for j in jobs]
  File "/Users/jieru/opt/anaconda3/envs/pytest36/lib/python3.6/site-packages/submitit/core/core.py", line 286, in results
    outcome, result = self._get_outcome_and_result()
  File "/Users/jieru/opt/anaconda3/envs/pytest36/lib/python3.6/site-packages/submitit/core/core.py", line 366, in _get_outcome_and_result
    f"Job {self.job_id} (task: {self.task_id}) with path {self.paths.result_pickle}\n"
submitit.core.utils.UncompletedJobError: Job 81029 (task: 0) with path /Users/jieru/workspace/hydra-fork/hydra/plugins/hydra_submitit_launcher/example/multirun/2020-05-29/10-35-45/.local/81029_0_result.pkl
has not produced any output (state: FINISHED)
Error stream produced:
----------------------
submitit ERROR (2020-05-29 10:35:46,642) - Could not dump error:
Can't pickle typing.Dict[str, str]: it's not the same object as typing.Dict

because of A temporary saved file already exists.
submitit ERROR (2020-05-29 10:35:46,643) - Submitted job triggered an exception
Traceback (most recent call last):
  File "/Users/jieru/opt/anaconda3/envs/pytest36/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Users/jieru/opt/anaconda3/envs/pytest36/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/jieru/opt/anaconda3/envs/pytest36/lib/python3.6/site-packages/submitit/core/_submit.py", line 11, in <module>
    submitit_main()
  File "/Users/jieru/opt/anaconda3/envs/pytest36/lib/python3.6/site-packages/submitit/core/submission.py", line 65, in submitit_main
    process_job(args.folder)
  File "/Users/jieru/opt/anaconda3/envs/pytest36/lib/python3.6/site-packages/submitit/core/submission.py", line 58, in process_job
    raise error
  File "/Users/jieru/opt/anaconda3/envs/pytest36/lib/python3.6/site-packages/submitit/core/submission.py", line 49, in process_job
    utils.pickle_dump(("success", result), tmppath)
  File "/Users/jieru/opt/anaconda3/envs/pytest36/lib/python3.6/site-packages/submitit/core/utils.py", line 278, in pickle_dump
    pickle.dump(obj, ofile, pickle.HIGHEST_PROTOCOL)
_pickle.PicklingError: Can't pickle typing.Dict[str, str]: it's not the same object as typing.Dict
@jieru-hu jieru-hu added the bug Something isn't working label May 29, 2020
jieru-hu added a commit to jieru-hu/hydra that referenced this issue May 29, 2020
@jieru-hu
Remove python 3.6
97b4b51 
facebookresearch#624
@omry
Copy link
Collaborator

omry commented May 29, 2020

This is a bug that was fixed in upstream cloud-pickle as far as I know.
can you try to upgrade it?

@jieru-hu
Copy link
Contributor Author

the exact exception happened while pickling (not cloudpickling)

https://github.com/facebookincubator/submitit/blob/master/submitit/core/utils.py#L278

def pickle_dump(obj: Any, filename: Union[str, Path]) -> None:
    with open(filename, "wb") as ofile:
        pickle.dump(obj, ofile, pickle.HIGHEST_PROTOCOL)

related pickle PR python/typing#511

@omry
Copy link
Collaborator

omry commented May 29, 2020

okay. regardless - I think submitit is dumping with cloudpickle.

@jrapin
Copy link
Contributor

jrapin commented Jun 2, 2020

Yes submitit is dumping with cloudpickle.
I'm pretty sure this error comes from trying to pickle a Dict[str, str] and not a dict:
if you instantiate a dict this way my_dict = Dict[str, str]() instead of my_dict: Dict[str, str] = {} then the object won't be picklable.
I did not see this in the plugin though, can this happen in Omegaconf?
I'll check in submitit itself as well but since the executor is not an attribute of the Launcher I don't think that the problem could come from it.

@jrapin
Copy link
Contributor

jrapin commented Jun 2, 2020

Couldn't find any in OmegaConf nor submitit, I think this may be due to ObjectConf though, since it inherits from Dict[str, str] and is used as default initializer

@omry
Copy link
Collaborator

omry commented Jun 2, 2020

since it inherits from Dict[str, str]

It does not .

This is still likely somewhere in there though. It keeps types around.

@jrapin
Copy link
Contributor

jrapin commented Jun 2, 2020 •
edited
Loading

it does not

I'm talking about ObjectConf

hydra/hydra/types.py

Line 13 in 2e8a381

class ObjectConf(Dict[str, Any]):

@omry
Copy link
Collaborator

omry commented Jun 3, 2020 •
edited
Loading 

$ pip freeze  | grep cloudpickle
cloudpickle==1.4.1

In [1]: from dataclasses import dataclass
In [2]: from typing import *
In [3]: @dataclass
   ...: class Foo(Dict[str, str]):
   ...:     n : int = 10
In [4]: import cloudpickle
In [5]: dump1 = cloudpickle.dumps(Foo())
In [6]: from omegaconf import OmegaConf
In [7]: dump2 = cloudpickle.dumps(OmegaConf.structured(Foo()))
In [8]: len(dump1)
Out[8]: 2052
In [9]: len(dump2)
Out[9]: 2426

@omry omry added the plugin Plugins realted issues label Jun 29, 2020
@omry
Copy link
Collaborator

omry commented Oct 19, 2020 •
edited
Loading

This does not seems to be worth the effort. We can revisit if anyone complains.
For now, the submitit plugin is officially Python 3.7+.

@omry omry closed this as completed Oct 19, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working plugin Plugins realted issues
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@omry @jieru-hu @jrapin