Multiple resetting connection after RPCError

[marcinh]

Description

When there is something wrong with communication master-worker (RPCError caught)), connection is being reset (new RPC server created) multiple times (actually after every heartbeat interval) until new message from worker appears. So when there is only single worker and RPCError is caught - connection is being reset infinitely.

Expected behavior

After catching RPCError connection should be reset once

Actual behavior

Connection is being reset every single second (heartbeat interval) until new message from worker arrives or new worker appears

Steps to reproduce

Setup locust master + single worker, trigger RPCError communication error somehow.

Possible solution

Currently connection_reset() method in MasterRunner is called in heartbeat_worker thread when connection_broken flag is set. Once this flag is set to True connection reset bill be done in a loop until this flag is set to false (what is done in client listener thread). So the possible solution is to move setting connection_broken flag to false into connection_reset() method itself

    def reset_connection(self):
        logger.info("Reset connection to worker")
        try:
            self.server.close()
            self.server = rpc.Server(self.master_bind_host, self.master_bind_port)
            self.connection_broken = False
        except RPCError as e:
            logger.error(f"Temporary failure when resetting connection: {e}, will retry later.")

Environment

• OS: Centos
• Python version: 3.9
• Locust version: 2.8.2

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions]

This issue was closed because it has been stalled for 10 days with no activity. This does not necessarily mean that the issue is bad, but it most likely means that nobody is willing to take the time to fix it. If you have found Locust useful, then consider contributing a fix yourself!