Parallelize absubmit

[lovesegfault]

Solves #2442

[sampsyo]

Looks like just what the doctor ordered! ✨ Would you mind adding a changelog entry?

[lovesegfault]

@sampsyo Almost! Trying to fix this right now

Traceback (most recent call last):
  File "./beet", line 23, in <module>
    beets.ui.main()
  File "/home/bemeurer/src/beets/beets/ui/__init__.py", line 1256, in main
    _raw_main(args)
  File "/home/bemeurer/src/beets/beets/ui/__init__.py", line 1243, in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "/home/bemeurer/src/beets/beetsplug/absubmit.py", line 110, in command
    pool.map(self.analyze_submit, items)
  File "/usr/lib64/python3.6/multiprocessing/pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib64/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
  File "/usr/lib64/python3.6/multiprocessing/pool.py", line 424, in _handle_tasks
    put(task)
  File "/usr/lib64/python3.6/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/lib64/python3.6/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: can't pickle _thread._local objects

Any ideas?

[sampsyo]

Hmm, not exactly. Maybe the problem is that multiprocessing is trying to pickle self to send it across the channel there, which could be causing problems? Maybe it’s possible to use pdb to find out what obj is here.

[lovesegfault]

Good idea, on it!

[lovesegfault]

@sampsyo Tested and ready for merging!

[sampsyo]

Hmm; threads do seem like the right thing here, but I’m slightly concerned that ThreadPool isn’t documented in the stdlib. Does this work on Python 2? If not, maybe we’d be better off with the thread pool in concurrent.futures?

[lovesegfault]

Oh, I don't have Python2 on my machine. Let me install it and test.

[lovesegfault]

It explodes with Python2

[sampsyo]

Got it. Well, it would be a perfectly defensible move to make this a Python-3-only feature and fall back to sequential execution on Python 2. If we do that, then the concurrent.futures route might be a more reliable route to get similar parallel map behavior here.

[lovesegfault]

@sampsyo I think I can make it works in parallel for both Python 2 and Python 3, but with slightly different techniques. I am giving it a shot right now :)

[lovesegfault]

@sampsyo I couldn't get it to work in Python 2, so I reverted to the nice original multithreding code for Python 3, and made it fallback to sequential on Python 2.

[sampsyo]

Looks fantastic; thank you for your help!