Multithreading support #1591
Merged
Multithreading support #1591
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The worker threads don't actually do anything, but with this change we can create and destroy threads roughly right. Towards IPERF-124.
While here, also fix a place where we forgot to join after cancel.
* Create iperf_send_mt and iperf_recv_mt, the multi-threaded versions of network I/O functions. These handle a single connection only and do not attempt to coordinate timing (for flow control) with any other threads. * Make client and server thread functions call the new multi-threaded network I/O functions. * Remove all network I/O for the test streams from the main thread. * fd_set objects in test object only apply to sockets used by the main thread (not the test streams in the worker threads). Outstanding issues: * No locking of shared data structures at this point. Correctness may be compromised at this point. * Worker threads on the sender side will tend to busy-wait because they do not attempt to sleep while attempting to pace themeselves. * No support (for now) for ending conditions other than time-based (packet-based and byte-based don't work).
We need to get the threads to exit when the ending conditions for -n / -k are reached, but we weren't doing that. While here, clean up some not-very-often-used error case code and pet copyright dates. Fixes IPERF-151.
Also do conditional compilation for stdatomic and pthread.
The issue is that the default compiler on CentOS 7 (and presumably RHEL 7) is an old GCC that's too old to support atomic types. In this case we attempt to approximate an atomic 64-bit type with a normal 64-bit integer, and warn at runtime if the system doesn't support lock-free operations with this data type. This has been compile-tested and lightly run-tested on CentOS 7 x86_64. It might not work well with ancient non-GCC compilers or 32-bit hosts.
This fixes a problem where every thread would essentially burn a CPU core busy-waiting, when it didn't need to. It's believed that this excess CPU usage might contribute to packet loss and poor performance. Non-blocking sockets were a necessity with the original single- thread process model of iperf3, in order to allow for concurrency between different test streams. With multiple threads, this is no longer necesary (it's perfectly fine for a thread to block on I/O, as it only services a single test stream and won't affect any others). Problem pointed out by @bltierney. Code review by @swlars. Fixes IPERF-177.
iperf3 implements a limit intended to allow the receiving side of a test to abort a test in progress if no data has been received for a certain length of time. This time limit is configured with the --rcv-timeout command-line option. The original implementation didn't work correctly with multi-threading because the code that implemented the limit had no visibility into the network I/O activity handled by other threads. The code has been restructured to make this work correctly, by watching the total number of blocks transferred in the test and using that to determine progress (or lack thereof). A minor change was also made to allow worker threads to be cancelled, even if they were blocked waiting for network I/O. While necessary for the testing protocol for this bug, this change might also improve the correctness of thread handling around the end of tests. Fixes IPERF-178.
We now handle the case where the worker threads exited on their own accord before the main thread had a chance to cancel them. While here, tweaked some of the error messages. Fixes IPERF-179.
ismailfaruk
added a commit
to ismailfaruk/iperf
that referenced
this pull request
May 3, 2024
commit 1843d37 Author: Bruce A. Mah <bmah@es.net> Date: Wed Mar 27 14:52:35 2024 -0700 Regen. commit d80b914 Author: David Bar-On <david.cdb004@gmail.com> Date: Sun Feb 18 20:07:30 2024 +0200 Add pthread missing functions in Android commit b5eb3a4 Author: David Bar-On <david.cdb004@gmail.com> Date: Sat Mar 23 09:48:03 2024 +0200 Fix issue 1662 - not allow negative test duration (with rebase and changes per reviewer comments) commit 46f6050 Author: Bruce A. Mah <bmah@es.net> Date: Fri Mar 22 08:53:49 2024 -0700 Don't chdir when becoming a daemon. This fixes some non-intuitive behavior when using the iperf3 authentication feature, where iperf3 was able to use a relative path to locate the credentials file when being run "normally" but not if it was being run as a --daemon (the workaround was to use only absolute pathname arguments). commit cd18344 Author: TheRealDJ <dj@djsnet.org> Date: Tue Mar 12 16:16:49 2024 -0400 Add stat reporting interval to JSON .start.test_start commit ed7d16c Author: Maks Mishin <maks.mishinFZ@gmail.com> Date: Fri Dec 15 00:13:48 2023 +0300 Fix descriptor leak in some return paths commit 111212b Author: Maks Mishin <maks.mishinFZ@gmail.com> Date: Thu Dec 14 23:46:46 2023 +0300 Fix memory leak with addrinfo commit e06177c Author: Maks Mishin <maks.mishinFZ@gmail.com> Date: Thu Dec 14 22:40:55 2023 +0300 Close the descriptor when forces exit commit 7b94705 Author: Bruce A. Mah <bmah@es.net> Date: Thu Mar 14 14:06:40 2024 -0700 Update copyright dates in selected places. commit b86ba29 Merge: c362e1a d7c021b Author: Bruce A. Mah <bmah@es.net> Date: Thu Mar 14 15:35:14 2024 -0400 Merge pull request esnet#1667 from esnet/bmah-fix-fqrate-server Properly reset the --fq-rate parameter on the server between tests. commit d7c021b Author: Bruce A. Mah <bmah@es.net> Date: Thu Mar 14 11:53:58 2024 -0700 Properly reset the --fq-rate parameter on the server between tests. Without this change, an --fq-rate setting would persist on the server, which could adversely slow down future --reverse tests. This bug was exposed by PR esnet#1643, which allows --fq-rate to work on the server. One annoying side-effect of this bug was that GitHub Actions scripts were timing out and throwing errors. Pet file copyright date while here. commit c362e1a Author: Bruce A. Mah <bmah@es.net> Date: Fri Feb 23 11:30:22 2024 -0800 Assume we always have inttypes.h and stdint.h (both from C99). This eliminates some compile-time tests that didn't really work as desired and aren't easy to fix. Inspired by comments on PR esnet#1636. commit e004dc3 Author: GBA <MrRegister@outlook.com> Date: Sat Feb 10 03:12:04 2024 +0800 Fix --rcv-timeout manual text (esnet#1642) * Fix --rcv-timeout manual text Correct the default value in ms from `12000` to `120000`. Source of truth: https://github.com/esnet/iperf/blob/master/src/iperf_api.h#L71 * Fix --rcv-timeout manual text commit 2c3d631 Author: Bruce A. Mah <bmah@es.net> Date: Tue Feb 6 10:47:05 2024 -0800 docs: Update docs to account for multi-threaded iperf3. commit d3e676c Author: Sarah Larsen <swlarsen@grobb3-dev-5.es.net> Date: Thu Jan 11 18:24:47 2024 +0000 Fix Issue esnet#1632 fq-rate works in reverse commit 0e90ae5 Author: Bruce A. Mah <bmah@es.net> Date: Thu Feb 8 10:45:34 2024 -0800 Regen. commit d4b2b31 Author: dopheide-esnet <dopheide@es.net> Date: Thu Feb 8 12:43:29 2024 -0600 minor error message correction for openssl includes (esnet#1649) * minor error message correction for openssl includes * update fix for config/ax_check_openssl.m4 commit 3510239 Merge: ec06f7b fd2b6d9 Author: Bruce A. Mah <bmah@es.net> Date: Wed Jan 3 18:23:15 2024 -0500 Merge pull request esnet#1613 from davidBar-On/affinity-help-text-correction Fix --affinity help text commit ec06f7b Merge: e4c4cc0 4264d09 Author: Bruce A. Mah <bmah@es.net> Date: Thu Dec 21 14:52:19 2023 -0500 Merge pull request esnet#1609 from kjander0/master Correct format specifier for printing int64_t commit e4c4cc0 Author: Christian Speich <christian@christian-speich.de> Date: Thu Dec 14 19:29:48 2023 +0100 Implement streaming json output (esnet#1098) Currently when enabling json output, the results are only written once the test concludes. This is done to output one full json document containing all relevant informations. To allow status output during the run while using json as output, this patch adds a newline-delimited JSON output. In order to achive this multiple event objects are emitted. These are serialized as json and printed with a newline seperating them. Each event contains a event name and its data. The following events have been introduced: start, interval, end, error, server_output_text and server_output_json. The data contains the relevant portion of the normal JSON output. Co-authored-by: swlars <89053414+swlars@users.noreply.github.com> commit e07eb70 Author: Bruce A. Mah <bmah@es.net> Date: Thu Dec 14 10:15:34 2023 -0800 Regen. commit f65178b Merge: cb84a76 1511e9f Author: Bruce A. Mah <bmah@es.net> Date: Thu Dec 14 13:13:30 2023 -0500 Merge pull request esnet#1612 from jpalus/link-libatomic Check and link libatomic if needed commit cb84a76 Merge: 9970208 b33373e Author: Bruce A. Mah <bmah@es.net> Date: Thu Dec 14 12:42:26 2023 -0500 Merge pull request esnet#1621 from esnet/remove-travis-ci-support Remove Travis CI support as we won't be using it going forward. commit b33373e Author: Bruce A. Mah <bmah@es.net> Date: Thu Dec 14 09:22:19 2023 -0800 Remove Travis CI support as we won't be using it going forward. commit 9970208 Author: Bruce A. Mah <bmah@es.net> Date: Wed Dec 6 16:50:03 2023 -0800 Regen. commit 8224827 Author: Bruce A. Mah <bmah@es.net> Date: Wed Dec 6 16:49:33 2023 -0800 Version number bump post 3.16. commit ecf8dd5 Merge: 402b48d 913f8df Author: Bruce A. Mah <bmah@es.net> Date: Wed Dec 6 19:35:32 2023 -0500 Merge pull request esnet#1595 from RayWindRiver/iperf_vxworks update to support VxWorks commit 913f8df Merge: 2dc942a 402b48d Author: Bruce A. Mah <bmah@es.net> Date: Wed Dec 6 19:18:20 2023 -0500 Merge branch 'master' into iperf_vxworks commit 402b48d Merge: a67fd67 9caff37 Author: Bruce A. Mah <bmah@es.net> Date: Wed Dec 6 18:55:07 2023 -0500 Merge pull request esnet#1468 from BrainSlayer/master fix endian type of udp connect / reply messages commit fd2b6d9 Author: David Bar-On <david.cdb004@gmail.com> Date: Tue Dec 5 09:58:57 2023 +0200 Fix --affinity help text commit 1511e9f Author: Jan Palus <jpalus@fastmail.com> Date: Sun Dec 3 12:14:05 2023 +0100 Check and link libatomic if needed Some architectures without native support for 64-bit atomics need linking with libatomic. commit a67fd67 Author: Bruce A. Mah <bmah@es.net> Date: Fri Dec 1 13:37:07 2023 -0800 Update for iperf-3.16. commit 4264d09 Author: kjander0 <kjander0@gmail.com> Date: Thu Nov 30 20:48:42 2023 +1000 Correct format specifier for printing int64_t. commit f9481e1 Author: Bruce A. Mah <bmah@es.net> Date: Wed Nov 29 11:46:13 2023 -0800 Regen. commit 0518055 Author: Bruce A. Mah <bmah@es.net> Date: Wed Nov 29 11:45:57 2023 -0800 Version number bumps for iperf-3.16. commit 97c2e40 Author: Bruce A. Mah <bmah@es.net> Date: Wed Nov 15 11:18:49 2023 -0800 Regen. commit eaada2b Author: Bruce A. Mah <bmah@es.net> Date: Wed Nov 15 11:18:05 2023 -0800 Bump version numbers for iperf-3.16-beta1. commit 2dc942a Author: Lei Liu <Lei.Liu@windriver.com> Date: Wed Nov 15 18:26:39 2023 +0800 update to support VxWorks commit 4bea03e Author: Bruce A. Mah <bmah@es.net> Date: Wed Nov 8 13:26:55 2023 -0800 Regen. commit b3686c6 Author: Bruce A. Mah <bmah@es.net> Date: Wed Nov 8 13:26:02 2023 -0800 Documentation cleanups and updates. Update to a sane version number. commit a393df1 Merge: ca7c875 291c48e Author: Bruce A. Mah <bmah@es.net> Date: Wed Nov 8 14:38:07 2023 -0500 Merge pull request esnet#1591 from esnet/mt Multithreading support commit 291c48e Author: Bruce A. Mah <bmah@es.net> Date: Wed Nov 8 11:20:21 2023 -0800 Regen. commit 6497aad Author: Sarah Larsen <swlarsen@es.net> Date: Fri Sep 29 19:04:00 2023 +0000 Add dates to RELNOTES.md for iperf-3.15-mt-beta1. commit 418eef5 Author: Bruce A. Mah <bmah@es.net> Date: Fri Sep 15 16:10:56 2023 -0700 Regen. commit 6cfec70 Author: Bruce A. Mah <bmah@es.net> Date: Fri Sep 15 16:09:40 2023 -0700 Version update for mt branch. commit 83a92d6 Author: Bruce A. Mah <bmah@es.net> Date: Fri Sep 15 16:08:33 2023 -0700 Add accumulated release notes from prior iperf-mt public beta releases. commit 83693ee Author: Sarah Larsen <swlarsen@grobb3-dev-5.es.net> Date: Mon May 8 14:37:06 2023 -0700 Fix debug level commit a40c9b3 Author: Bruce A. Mah <bmah@es.net> Date: Fri Apr 28 10:23:12 2023 -0700 Make thread shutdown more tolerant. We now handle the case where the worker threads exited on their own accord before the main thread had a chance to cancel them. While here, tweaked some of the error messages. Fixes IPERF-179. commit 2e80758 Author: Bruce A. Mah <bmah@es.net> Date: Thu Apr 27 18:28:06 2023 -0700 Fix a bug related to idle timeouts on the test receiver. iperf3 implements a limit intended to allow the receiving side of a test to abort a test in progress if no data has been received for a certain length of time. This time limit is configured with the --rcv-timeout command-line option. The original implementation didn't work correctly with multi-threading because the code that implemented the limit had no visibility into the network I/O activity handled by other threads. The code has been restructured to make this work correctly, by watching the total number of blocks transferred in the test and using that to determine progress (or lack thereof). A minor change was also made to allow worker threads to be cancelled, even if they were blocked waiting for network I/O. While necessary for the testing protocol for this bug, this change might also improve the correctness of thread handling around the end of tests. Fixes IPERF-178. commit 77685c1 Author: Bruce A. Mah <bmah@es.net> Date: Fri Apr 21 14:56:34 2023 -0700 Update description of -P option for multithreading. commit 6e75b07 Author: Bruce A. Mah <bmah@es.net> Date: Wed Apr 19 16:21:35 2023 -0700 Don't set O_NONBLOCK on sockets used for tests. This fixes a problem where every thread would essentially burn a CPU core busy-waiting, when it didn't need to. It's believed that this excess CPU usage might contribute to packet loss and poor performance. Non-blocking sockets were a necessity with the original single- thread process model of iperf3, in order to allow for concurrency between different test streams. With multiple threads, this is no longer necesary (it's perfectly fine for a thread to block on I/O, as it only services a single test stream and won't affect any others). Problem pointed out by @bltierney. Code review by @swlars. Fixes IPERF-177. commit 20a02b4 Author: Bruce A. Mah <bmah@kitchenlab.org> Date: Thu Apr 13 16:04:18 2023 -0700 Fix build on CentOS 7 / x86_64. The issue is that the default compiler on CentOS 7 (and presumably RHEL 7) is an old GCC that's too old to support atomic types. In this case we attempt to approximate an atomic 64-bit type with a normal 64-bit integer, and warn at runtime if the system doesn't support lock-free operations with this data type. This has been compile-tested and lightly run-tested on CentOS 7 x86_64. It might not work well with ancient non-GCC compilers or 32-bit hosts. commit a326ec8 Author: Bruce A. Mah <bmah@es.net> Date: Thu Apr 13 14:43:39 2023 -0700 Add autoconf test for stdatomic Also do conditional compilation for stdatomic and pthread. commit b14c3b9 Author: Bruce A. Mah <bmah@es.net> Date: Wed Mar 29 18:52:33 2023 -0700 Make -n / -k options work correctly. We need to get the threads to exit when the ending conditions for -n / -k are reached, but we weren't doing that. While here, clean up some not-very-often-used error case code and pet copyright dates. Fixes IPERF-151. commit 30ce2d5 Author: Sarah Larsen <swlarsen@es.net> Date: Tue Jan 17 17:44:27 2023 -0800 Add atomic_iperf_size_t for atomic (thread-safe) operations commit 0755cc4 Author: Bruce A. Mah <bmah@es.net> Date: Fri Oct 28 11:52:25 2022 -0700 Initial version of multi-threaded writers. * Create iperf_send_mt and iperf_recv_mt, the multi-threaded versions of network I/O functions. These handle a single connection only and do not attempt to coordinate timing (for flow control) with any other threads. * Make client and server thread functions call the new multi-threaded network I/O functions. * Remove all network I/O for the test streams from the main thread. * fd_set objects in test object only apply to sockets used by the main thread (not the test streams in the worker threads). Outstanding issues: * No locking of shared data structures at this point. Correctness may be compromised at this point. * Worker threads on the sender side will tend to busy-wait because they do not attempt to sleep while attempting to pace themeselves. * No support (for now) for ending conditions other than time-based (packet-based and byte-based don't work). commit fc86f85 Author: Bruce A. Mah <bmah@es.net> Date: Thu Aug 4 23:10:12 2022 -0700 Locking around output operations. Allow threads to exit gracefully. commit 0b1be25 Author: Bruce A. Mah <bmah@es.net> Date: Thu Aug 4 18:37:18 2022 -0700 Improve error handling around thread attribute calls. commit a360f70 Author: Bruce A. Mah <bmah@es.net> Date: Thu Aug 4 11:19:41 2022 -0700 Improve error handling. While here, also fix a place where we forgot to join after cancel. commit 6bcbb20 Author: Bruce A. Mah <bmah@es.net> Date: Mon Jul 25 18:50:40 2022 -0700 Minor configure.ac fixes (remove AC_PROG_RANLIB, only LT_INIT once, fix typo). commit b4c23ab Author: Bruce A. Mah <bmah@es.net> Date: Mon Jul 25 18:46:42 2022 -0700 Add autoconf support for POSIX threads. commit 448ba70 Author: Bruce A. Mah <bmah@es.net> Date: Fri Jul 22 14:47:15 2022 -0700 Add (nonfunctional) worker thread per stream. The worker threads don't actually do anything, but with this change we can create and destroy threads roughly right. Towards IPERF-124. commit ca7c875 Author: Bruce A. Mah <bmah@es.net> Date: Fri Oct 27 16:04:02 2023 -0400 Bmah openssl3 (esnet#1589) WIP for OpenSSL 3.0 support. We attempt to migrate off of APIs that are deprecated in OpenSSL 3.0, as detected by using OPENSSL_VERSION_MAJOR. In the case of sha256(), a series of deprecated library calls was replaced by a single macro that is supported on both OpenSSL 1.x and 3.x. Fixes esnet#1300. commit a9a55e4 Author: David Bar-On <61089727+davidBar-On@users.noreply.github.com> Date: Mon Oct 23 21:44:43 2023 +0300 Per esnet#1583 add check that server authorized users file is accessible (esnet#1585) commit 9caff37 Author: Sebastian Gottschall <s.gottschall@dd-wrt.com> Date: Tue Feb 7 12:59:55 2023 +0600 fix endian type of udp connect / reply messages for example. if you run a iperf3 server on a little endian host, but client is bug endian it will result in the following error if you try to start a udp bandwidth test Connect received for Socket 5, sz=4, buf=36373839, i=0, max_len_wait_for_reply=4 iperf3: error - unable to read from stream socket: Invalid argument the reason that that all messages are reversed. fix this by swapping the message values to little endian host order on big endian systems Signed-off-by: Sebastian Gottschall <s.gottschall@dd-wrt.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Version of iperf3 (or development branch, such as
masteror3.1-STABLE) to which this pull request applies:masterIssues fixed (if any):
Brief description of code changes (suitable for use as a commit message):
Support the use of multiple threads for parallel tests. Requires pthread and stdatomic support.