stripped tasks can't be pickled

[tomsilver]

to reproduce:

from predicators.envs import create_new_env
from predicators import utils
import dill as pkl

utils.reset_config({"num_train_tasks": 1})
env = create_new_env("blocks", do_cache=True)
train_tasks = env.get_train_tasks()
stripped_task = utils.strip_task(train_tasks[0], env.predicates)
with open("/tmp/test.pkl", "wb") as f:
    pkl.dump(stripped_task, f)
with open("/tmp/test.pkl", "rb") as f:
    pkl.load(f)

Leads to:

Traceback (most recent call last):
  File "/Users/tom/phd/predicators/predicators/predicators_wtf.py", line 12, in <module>
    pkl.load(f)
  File "/Users/tom/.pyenv/versions/3.10.6/lib/python3.10/site-packages/dill/_dill.py", line 272, in load
    return Unpickler(file, ignore=ignore, **kwds).load()
  File "/Users/tom/.pyenv/versions/3.10.6/lib/python3.10/site-packages/dill/_dill.py", line 419, in load
    obj = StockUnpickler.load(self)
  File "/Users/tom/phd/predicators/predicators/structs.py", line 337, in __hash__
    return self._hash
  File "/Users/tom/.pyenv/versions/3.10.6/lib/python3.10/functools.py", line 981, in __get__
    val = self.func(instance)
  File "/Users/tom/phd/predicators/predicators/structs.py", line 320, in _hash
    return hash(str(self))
  File "/Users/tom/phd/predicators/predicators/structs.py", line 323, in __str__
    return self._str
  File "/Users/tom/.pyenv/versions/3.10.6/lib/python3.10/functools.py", line 981, in __get__
    val = self.func(instance)
  File "/Users/tom/phd/predicators/predicators/structs.py", line 385, in _str
    return (str(self.predicate) + "(" + ", ".join(map(str, self.objects)) +
AttributeError: 'GroundAtom' object has no attribute 'predicate'

[tomsilver]

even more minimal:

from predicators.envs import create_new_env
from predicators import utils
from predicators.structs import Task, DefaultState
import dill as pkl

utils.reset_config({"num_train_tasks": 1})
env = create_new_env("blocks", do_cache=True)
train_tasks = env.get_train_tasks()
task = train_tasks[0]
stripped_task = Task(DefaultState, set(task.goal))
with open("/tmp/test.pkl", "wb") as f:
    pkl.dump(stripped_task, f)
with open("/tmp/test.pkl", "rb") as f:
    pkl.load(f)

where the key thing is set(task.goal). using just task.goal doesn't error. very weird.

[tomsilver]

circumventing tasks completely, still crashes!

from predicators.envs import create_new_env
from predicators import utils
import dill as pkl

utils.reset_config({"num_train_tasks": 1})
env = create_new_env("blocks", do_cache=True)
train_tasks = env.get_train_tasks()
task = train_tasks[0]
stripped_goal = set(task.goal)
with open("/tmp/test.pkl", "wb") as f:
    pkl.dump(stripped_goal, f)
with open("/tmp/test.pkl", "rb") as f:
    pkl.load(f)

[tomsilver]

this one doesn't crash, so I guess there is some difference between the train tasks created by the environment and the one that I'm creating here?

from predicators.structs import Predicate, Type, State, Task
import dill as pkl

type1 = Type("type1", ["feat1", "feat2"])
obj3 = type1("obj3")
obj7 = type1("obj7")
state = State({
    obj3: [1, 2],
    obj7: [3, 4],
})
cup_type = Type("cup_type", ["feat1"])
plate_type = Type("plate_type", ["feat1"])
pred = Predicate("On", [cup_type, plate_type], lambda s, o: True)
pred2 = Predicate("On", [cup_type, plate_type], lambda s, o: False)
cup = cup_type("cup")
plate = plate_type("plate")
goal = {pred([cup, plate])}
task = Task(state, goal)

stripped_goal = set(task.goal)
with open("/tmp/test.pkl", "wb") as f:
    pkl.dump(stripped_goal, f)
with open("/tmp/test.pkl", "rb") as f:
    pkl.load(f)

[tomsilver]

this is some pretty weird shit, I really don't know... @ronuchit want a Python puzzle?

from predicators.envs import create_new_env
from predicators import utils
import dill as pkl

utils.reset_config({"num_train_tasks": 1})
env = create_new_env("blocks", do_cache=False)
train_tasks = env.get_train_tasks()
task = train_tasks[0]
bad_atom = next(iter(task.goal))

block1 = env._block_type("block2")
block2 = env._block_type("block1")
fine_atom = env._On([block1, block2])
_ = hash(fine_atom)

# Succeeds
assert fine_atom == bad_atom

# Nothing discernably different
for field in dir(fine_atom):
    print(field)
    print("FINE:")
    print(getattr(fine_atom, field))
    print("BAD:")
    print(getattr(bad_atom, field))

# Succeeds
with open("/tmp/test.pkl", "wb") as f:
    pkl.dump(fine_atom, f)
with open("/tmp/test.pkl", "rb") as f:
    pkl.load(f)

# Fails on load (AttributeError: 'GroundAtom' object has no attribute 'predicate')
with open("/tmp/test.pkl", "wb") as f:
    pkl.dump(bad_atom, f)
with open("/tmp/test.pkl", "rb") as f:
    pkl.load(f)

[ronuchit]

i do not know sorry! it really depends on what's happening inside get_train_tasks

[tomsilver]

@ashayathalye ran into a similar issue and realized that it's likely related to some of these:

• https://stackoverflow.com/questions/54679349/cannot-unpickle-some-data-just-pickled-with-python-3-with-a-cycle
• https://stackoverflow.com/questions/44887394/pickle-dill-cannot-handle-circular-references-if-hash-is-overridden
• https://bugs.python.org/issue1761028