Deadlock when accessing attribute from one netref in another.

[SergeyYanos]

If you have two netrefs on two connections and try to access attributes of one from the other you will get deadlock in protocol.sync_recv_and_dispatch method.

From what I was able to gather:
When we first call run_example from t1(thread 1) we enter protocol.sync_recv_and_dispatch of e1(Example obj 1) and take the _sync_lock, in addition t2 takes the _sync_lock of e2. Then t1 tries to take _sync_lock of e2 when it is accessing e2.exposed_e meanwhile t2 is doing the same with _sync_lock of e1 and e1.exposed_e. This results in deadlock because t1 holds a lock that t2 wants and won't release it until it(t1) gets the lock that t2 holds which in turn waits for t1 to release the lock it(t2) wants.

Environment

• rpyc version=3.4.4
• python version=2.7
• operating system=Linux(Fedora 27).

Minimal example(sketchbook.py)

import rpyc
import logging
import threading


logging.basicConfig(level=logging.DEBUG, format='%(levelname)-8s : %(name)-10s - %(message)s')


class Example(object):
    def __init__(self, e):
        self.exposed_e = e
        self.logger = logging.getLogger('example_%s' % e)

    def run_example(self, examples):
        for _ in range(1000):
            for example in examples:
                self.logger.debug('%s accessing %s', self.exposed_e, example.exposed_e)


def main():
    logger = logging.getLogger('main')
    logger.debug('Starting test')

    host1 = '10.135.174.5'
    host2 = '10.135.175.5'
    port = '18812'

    conn1 = rpyc.classic.connect(host1, port)
    conn2 = rpyc.classic.connect(host2, port)

    examples = [conn1.modules['sketchbook'].Example(1),
                conn2.modules['sketchbook'].Example(2)]

    threads = [threading.Thread(target=e.run_example, args=(examples,)) for e in examples]

    for t in threads:
        t.start()
    logger.debug('Threads started!')

    for t in threads:
        t.join()
    logger.debug('Test finished!')


if __name__ == "__main__":
    main()

[coldfix]

Hi,

thanks for reporting and analyzing the issue.

I'm not surprised that this happens. This can probably be fixed by dropping _sync_lock/_sync_event and using only _recvlock instead. However, before doing this, I will have to think about it more carefully, (because I don't fully understand the reason why sync_lock is there at all right now).

Honestly, the code base looks in many places like it was originally written with single-threaded applications in mind and the support for multithreading was only added later. This leads to some architectural deficits with many edge cases. If possible, I would recommend not sharing the same connection across multiple threads and in particular avoiding communication between two shared connections. In fact, I believe the request/reactor mechanism is broken in many places and the only real fix would be to have a reactor that runs every request in its own thread (or better greenlet for performance reasons), i.e. migrating rpyc to fully take advantage of gevent.

That being said, I will of course try to fix every issue that can be identified and easily fixed.

[coldfix]

Hi, pushed a commit that fixes this particular issue (won't be released until in a few weeks though). Let me know if you encounter further problems. Best, Thomas