Random disconnect during transmission over WiFi

[osrf-migration]

Original report (archived issue) by Bart Cox (Bitbucket: bcox_pv).

---

Description

When we use ignition transport over WiFi we experience long delays on communication via (asynchronous) service calls and disconnects on pub/sub traffic. These seem to be accompanied with frequent detected disconnects and connects in the discovery layer. Interestingly, these delays seem to happen to a few nodes (but not all) at once and seem to resolve at the same time as well. We have been able to rule out any deadlock-like situations as our nodes will still accept and process service requests from nodes not affected by the delay in the network. Once the delay resolves, the messages seem to come in all at once.

We tested this problem with the basic example code from the source. When running the basic examples publisher.cc and subscriber.cc over WiFi, random disconnection callbacks are fired while both machines are still connected to the same network. We seem to experience similar problems with communication in the publisher/subscriber example which disconnects within a few minutes and in severe cases even seconds.

To rule out relevant external factors, we used an isolated network without any other active clients on a professional grade router and access-point but that seemed to have no influence on the robustness of the connections. We have also been able to exclude Ubuntu versions (16.04/18.04), client hardware/architecture and ignition-transport versions(5.xx - 7.xx), during our tests.

When we run the same tests on the same machines over a wired network no long delays or disconnects are occurring, the connection is stable.

Steps to Reproduce

• Use Ubuntu 18.04
• Install dependencies

sudo apt-get update\
          && apt-get -y install\
            gnupg lsb-release\
            cmake pkg-config cppcheck git mercurial build-essential curl\
            libprotobuf-dev protobuf-compiler libprotoc-dev libzmq3-dev uuid-dev\
            doxygen ruby-ronn libsqlite3-dev g++-8\
          && update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-8 800 --slave /usr/bin/g++ g++ /usr/bin/g++-8 --slave /usr/bin/gcov gcov /usr/bin/gcov-8
echo "deb <http://packages.osrfoundation.org/gazebo/ubuntu-stable> $(lsb_release -cs) main" > /etc/apt/sources.list.d/gazebo-stable.list
echo "deb <http://packages.osrfoundation.org/gazebo/ubuntu-prerelease> `lsb_release -cs` main" > /etc/apt/sources.list.d/gazebo-prerelease.list
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys D2486D2DD83DB69272AFE98867170598AF249743

• Install ignition libraries

sudo apt-get update \
          && sudo apt-get -y install \
            libignition-cmake2-dev \
            libignition-math6-dev \
            libignition-msgs4-dev \
            libignition-tools-dev

• Build ignition-transport 7.7.0 from source

‌

hg clone https://osrf-migration.github.io/ignition-gh-pages/#!/ignitionrobotics/ign-transport/ ign-transport \ 
&& cd ign-transport \ 
&& hg up ignition-transport7_7.0.0 \ 
&& mkdir -p build \ 
&& cd build \ 
&& cmake ../ \ 
&& make -j4 \ 
&& sudo make install -j4 \ 
&& cd ../example \ 
&& mkdir -p build \ 
&& cd build \ 
&& cmake .. \ 
&& make -j4

• Run the publisher example from the source code on machine A

‌

export IGN_PARTITION=transmission_test 
export IGN_VERBOSE=1 
export IGN_IP=${OWN_IP} 
./build/publisher

• Run the subscriber example from the source code on machine B

‌

export IGN_PARTITION=transmission_test 
export IGN_VERBOSE=1 
export IGN_IP=${OWN_IP} 
./build/subscriber

Expected behavior:

No disconnection callbacks when the machine is connected to the (wireless) network

Actual behavior:

After 2 minutes the subscriber gets a disconnect callback and stops receiving messages. The publisher keeps sending messages.

Reproduces how often:

Periodically.

Versions

• Ubuntu 18.04
• source install
• ignition-transport 7.7.0

Additional context

Our first assumption was that UDP multicast traffic carrying discovery information might get lost over a WiFi connection. Therefore we have been experimenting with different parameter sets in the discovery layer such as a lower heartbeat interval, higher silence interval etc. Only a longer silence interval resulted in a better performance in our tests but only at large values of 20 seconds or more.

We have further tried forcing all the traffic over unicast through modifying the relay functionality such that all discovery related messages are send over unicast within the same network (but not relayed). We were hoping that this lead to more stable connections but we did not see any significant improvement.

[osrf-migration]

Original comment by Bart Cox (Bitbucket: bcox_pv).

---

• Edited issue description

[osrf-migration]

Original comment by Nate Koenig (Bitbucket: Nathan Koenig).

---

• set assignee_account_id to "557058:095b1e12-74ed-4e20-b44f-2f0745b616e0"
• set assignee to "nkoenig (Bitbucket: nkoenig, GitHub: nkoenig)"

[osrf-migration]

Original comment by Nate Koenig (Bitbucket: Nathan Koenig).

---

Thanks for the information. I'll work on reproducing your test setup. In the mean-time, can you try out pull request #416? That PR feels related to this issue, but it's a shot in the dark right now.

[osrf-migration]

Original comment by Bart Cox (Bitbucket: bcox_pv).

---

I’ve run the revision of pull request #416 but the same problems persists on WiFi.

The bench test finishes successfully but when running a publisher/subscriber example, all transmissions halts on average after 250 seconds (sample size is 20).

For completeness I ran the same code on a fully wired connection and no errors were encountered (sample size is again 20 and the tests were stopped after 15 minutes due to the lack of any errors).

‌

[osrf-migration]

Original comment by Nate Koenig (Bitbucket: Nathan Koenig).

---

Okay. Thanks for the info. I'm still digging into the problem.

[osrf-migration]

Original comment by Nate Koenig (Bitbucket: Nathan Koenig).

---

• changed assignee_account_id from "557058:095b1e12-74ed-4e20-b44f-2f0745b616e0" to "557058:4ded1ddf-947e-4154-bbd1-3dba24f1bdbd"
• changed assignee from "nkoenig (Bitbucket: nkoenig, GitHub: nkoenig)" to "caguero (Bitbucket: caguero, GitHub: caguero)"

[osrf-migration]

Original comment by Carlos Agüero (Bitbucket: caguero, GitHub: caguero).

---

See pull request #436, Bart Cox (bartcox) , could you confirm that the pull request fixes your issue (only for pub/sub for now)?

[nkoenig]

Closing in order to triage the issue tracker. Please re-open if the problem persists.