pickle (as opposed to cPickle) support

[Guillaume227]

Hi everyone,

Could someone shed some light on the reason why pybind11 doesn't support pickle (as opposed to cPickle)?
Is there a hard reason or can it be worked on, in which case, could you provide some pointers?

I am migrating a project from boost::python to pybind11.
That project is using the flexibility that pickle (as opposed to cPickle) offers to substitute class/module/package names at unpickle time.
The idea is to unpickle transparently classes that have been renammed / moved and there is a decent number of pickled filed I'd rather not have to touch to be able to load them with pybind11.

`
def mapped_load_global(self):
module = mapname(self.readline()[:-1])
name = mapname(self.readline()[:-1])
klass = self.find_class(module, name)
self.append(klass)

unpickler = pickle.Unpickler(fileObj)
unpickler.dispatch[pickle.GLOBAL] = mapped_load_global
`
(can't get the code formating to work sorry)

[wjakob]

The reason is simple: Python 2.7 pickle calls the pybind11 internal initializers in an incorrect order, which leads to a crash, while cPickle doesn't. For Python 3, there is only pickle, so everything works by default.

I won't have time to chase down why Python 2.7 pickle does it this way, and if there is a way to avoid the segfault (as Python 2.7 is deprecated in any case). Patches are welcomed though (it sounds like this is important to you).

[Guillaume227]

I wasn't aware of that 2.7 pickle behavior. After weighing my options I have actually taken this opportunity to convert my project to python 3. Thank you for the great work.

[shartte]

I ran into this problem and since our project is unable to move on from Python 2.7, I solved this by implementing the reduce function as follows:

py::tuple reduce() {
	auto constructor = py::cast(*this).get_type();
	auto unpickleArgs = py::make_tuple();
	auto pickledValue = get_state(*this);

	return py::make_tuple(constructor, unpickleArgs, pickledValue);
}

There is probably a better way of getting the python type object though. But this works for me.

edit: Sadly it doesn't work mainly because of this line in pybind11.h:

rec->is_constructor = !strcmp(rec->name, "__init__") || !strcmp(rec->name, "__setstate__");

I am not sure why setstate is treated as a constructor in Python 2.7, pickle is simply calling it after construction and ignores the result value.