Issues with BPF packet scheduler #336
Comments
matttbe
changed the title
|
Hi @FoRward-999 Thank you for having reported these issues. The BPF packets schedulers are under development, see #75. It is good to have some feedback!
Do you mean the data are being re-sent? Or the MPTCP data fin?
Which application are you talking about here?
What other options are in this RST? A packet trace might be useful here. (I guess you don't have these issues with the default (non-BPF) packets schedulers, right?)
Good question. @geliangtang (who created these BPF packets schedulers with reviews from @mjmartineau ), any ideas where this could come from? Do you mind looking at this issue please? :)
I'm not sure to see the link with #52 (MP_FAIL support). Are you talking about this commit? 64b9cea ("mptcp: fix spurious retransmissions") which contains This is quite old, I don't think this is linked. |
|
Hello @matttbe, thank you for your reply. First of all, sorry for the late response.
Yes, the data are being re-sent. The packet trace on the client side is linked here: It seems that after the subflows are established, the server returns TCP ACK instead of MPTCP ACK. We didn't spot this problem before.
We use a simple single-threaded loop sending program written by ourselves. To avoid ambiguity, we did the same experiments using iperf3 with mptcpize. On the server side, we did: mptcpize run iperf3 -sOn the client side, we did: mptcpize run iperf3 -c 192.168.2.31 -n 1GBWe encountered problem 2 and problem 3 which were mentioned before. Problem 2: Client side: iperf3 is stuck, and the command forward@forward-virtual-machine:~/CODING/WorkSpace/Scheduler$ ps -ef |grep iperf3
forward 2542 1 99 17:03 ? 00:03:02 iperf3 -c 192.168.2.31 -n 1GB
forward 2673 2336 0 17:06 pts/1 00:00:00 grep --color=auto iperf3Server side: forward@forward-virtual-machine:~$ mptcpize run iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.2.30, port 57528
[ 5] local 192.168.2.31 port 5201 connected to 192.168.2.30 port 57542
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 128 KBytes 1.05 Mbits/sec
[ 5] 1.00-2.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 2.00-3.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 3.00-4.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 4.00-5.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 5.00-6.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 6.00-7.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 7.00-8.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 8.00-9.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 9.00-10.00 sec 0.00 Bytes 0.00 bits/secThe packet trace on the client side is linked here: Problem 3: Server side: forward@forward-virtual-machine:~$ mptcpize run iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 0:200:4956:0:400:0:fe7f:0, port 0
iperf3: error - unable to receive control message: Connection reset by peer
-----------------------------------------------------------The packet trace on the client side is linked here: According to packet traces above, the server returned TCP ACK instead of MPTCP ACK.
Yes, I don't have these issues with the default (non-BPF) packets schedulers. And the BPF packets schedulers run well when there is only one master flow.
Yes, this commit |
|
Hello @FoRward-999
Sorry, my turn.
I had a quick look and in the traces:
In other word (but maybe because I only quickly looked at the packet traces), it doesn't look like a packet scheduler issue. I don't understand why the receiver doesn't include the MPTCP options. Maybe a fallback has been done and the BPF packet scheduler are not handling that properly? It could be good to look at the MPTCP counters with I guess this was with your custom program, right? Maybe it is easy for you to track if the sender got the notification the receiver side has closed the connection? And check what the the receiver is doing after the close?
Indeed, surprising.
Here, we can see the MPTCP options in the ACKs. @geliangtang did you experiment such issues when doing your tests?
Indeed, strange. We might need
OK, good to know. FYI, we ended up discussion more about the packet scheduler API (and the behaviour of the default one) and more changes are needed: #350 (quite a bit... if you are interested ;-) )
👍 |
Yes, I think it's related to the BPF packet scheduler. I did as you mentioned. Here is the result of the #kernel
MPTcpExtMPCapableSYNRX 1 0.0
MPTcpExtMPCapableACKRX 1 0.0
MPTcpExtMPJoinSynRx 3 0.0
MPTcpExtMPJoinAckRx 3 0.0
MPTcpExtOFOQueueTail 2 0.0
MPTcpExtOFOQueue 3 0.0
MPTcpExtEchoAdd 1 0.0
MPTcpExtRcvPruned 77 0.0
MPTcpExtRcvWndShared 1 0.0
Yes, it's our custom program. The receiver sends a FIN after ~10 sec because we manually close the sever through The sender didn't get the notification the receiver side has closed the connection.
After several tries with iperf3 is stuck, as well as the whole client virtual machine. Here is the result of the #kernel
MPTcpExtMPCapableSYNRX 2 0.0
MPTcpExtMPCapableACKRX 2 0.0
MPTcpExtMPTCPRetrans 1 0.0
MPTcpExtMPJoinSynRx 2 0.0
MPTcpExtMPJoinAckRx 1 0.0
MPTcpExtDuplicateData 1 0.0
MPTcpExtEchoAdd 2 0.0
MPTcpExtRcvWndShared 1 0.0Due to the result of
Thank you for reminding. |
|
Hi, I'm currently trying to set up MPTCP in a virtual machine with Ubuntu 22.04 and Linux kernel 6.1.18. How can I select the scheduler that I want? When I compile the kernel I don't see the option to select it and neither does the net environment variable mptcp.scheduler. I would like to test with the round robin scheduler (bpf_rr). Could you tell me how to do it?Maybe I have to compile the bpf module? |
Hello, I would be interested to know how you selected the scheduler you wanted when testing. If you could give some orientation |
|
Hi @AlejandraOliver, This feature is still in progress, see #75 (and currently blocked by #350). If you are interested by testing and improving it, feel free to compile the kernel using the source code from our repository, the There is no so much documentation for the moment. It is not ideal but info can be extracted from the new BPF selftests: https://github.com/multipath-tcp/mptcp_net-next/commits/export |
|
Hello @AlejandraOliver , To enable the bpf scheduler, you can follow these steps:
I hope this will help. |
|
Hi @FoRward-999 ,
I would like to be able to test with this scheduler in kernel 6.1 so I would be grateful if you could tell me any orientation of to solve it. Thanks. |
@AlejandraOliver : it can only work with a kernel compiled from our |
|
The kernel of your export branch is v6.3? |
It is on top of the net-next branch which is on top of Linus branch. |
|
I have tried with the kernel of your export branch, I have compiled it and then, compiled the mptcp_bpf_rr.c with the clang tool. This creates an object file (mptcp_bpf_rr.o). Now, how do you load it in the kernel? I can´t use the bpftool tool and when I try to do it with 'insmod' I received errors. Thanks. |
Why? Please share the commands that you used and the output. Please share that in text and not in a screenshot. Also, what's the output of:
@geliangtang / @FoRward-999 do you mind adding the commands that you used to compile and load one of the eBPF MPTCP Packet scheduler program please? |
|
$ bpftool prog load /home/test2/mptcp_net-next/tools/testing/selftests/bpf/progs/mptcp_bpf_rr.o /sys/fs/bpf/mptcp_bpf_rr WARNING: bpftool not found for kernel 6.3.0 The output of the other commands are these: $ uname -a $ sudo dmesg | grep -i mptcp $ sysctl net.mptcp When I try to download some of the packages suggested for bpftool it tells me that they don't exist (I downloaded the kernel from your export branch). |
|
@AlejandraOliver if I'm not mistaken, Can you not see the MPTCP packet scheduler using: You should be able to set the new packet scheduler: To avoid the warnings with bpftool, you can compile bpftool, it should be easy, something like: Or follow instructions from: https://github.com/libbpf/bpftool |
|
when I try to compile bpftool to be able to load the mptcp_bpf_rr.o that I created with clang tool into the kernel, I received the following error: I I do $sudo bpftool prog list, I still received: |
How did you compile the kernel before? |
|
Hi @matttbe These create 4 packages .deb and I install them all and reboot the machine. One of the pacakges (dbg...) have errors but I don't think this is the problem. If I use these commands (make -j Maybe is the way I compile and install the kernel. What commands should I use to compile it instead of that ones? |
@AlejandraOliver mmh, strange, the error seems to suggest there is an issue with the kernel config or with cross compilation. It works for me with Maybe try to clean, re-compile the kernel, then bpftool? If it is not enough, the best is to look at https://github.com/libbpf/bpftool, try to compile bpftool from there and report issue there. |
|
@matttbe : Could you tell me which commands do you use to compile the kernel? Maybe that's the problem because I have enabled CONFIG_DEBUG_INFO_BTF (CONFIG_DEBUG_INFO_BTF=y) and before compile the kernel I have installed the required dependencies that are listed in Documentation/process/changes.rst. |
|
The best is probably to ask bpftool maintainers :-/ I compiled the kernel with You can also retry from scratch ( |
|
@FoRward-999 could you comment the commands that you use to compile and load the BPF packet scheduler in the kernel? Thanks in advance. |
|
Hello @AlejandraOliver |
|
Hi @FoRward-999, test2@test2-VirtualBox:~/mptcp_net-next/tools/testing/selftests/bpf/progs$ sudo clang -O2 -target -bpf -g -c mptcp_bpf_rr.c -o mptcp_bpf_rr.o In file included from mptcp_bpf_rr.c:4: I don't know why I get this error since I guess you didn't get it. The asm-types.h file is in the mptcp_net-next folder. How can I fix it? |
|
Try this: |
|
Thank you very much!! . This resolve the problem. When I do The thing is that if I do |
|
Yes, I think so.
Try: |
|
I've got this output when running Is that correct? So I can do the kselftest like mptcp_connect.sh and mptcp_join.sh and check it with tcpdump? |
|
Yes, it's correct. I didn't do the kselftest, you can give it a try. |
|
Ok, thanks!! |
|
Hi @FoRward-999 , were you able to fix the problem with bpf_rr?:
Server side: I have the same output in iperf3 when using this scheduler |
|
No, I still have this problem. |
|
Ok, thanks!! |
|
Hello @FoRward-999. When I execute the command The bpf_tcp_helpers.h is in ~/mptcp_net-next/tools/testing/selftests/bpf , How can i solve it? |
|
Hello @allen0091 |
Add a test case which replaces an active ingress qdisc while keeping the miniq in-tact during the transition period to the new clsact qdisc. # ./vmtest.sh -- ./test_progs -t tc_link [...] ./test_progs -t tc_link [ 3.412871] bpf_testmod: loading out-of-tree module taints kernel. [ 3.413343] bpf_testmod: module verification failed: signature and/or required key missing - tainting kernel #332 tc_links_after:OK #333 tc_links_append:OK #334 tc_links_basic:OK #335 tc_links_before:OK #336 tc_links_chain_classic:OK #337 tc_links_chain_mixed:OK #338 tc_links_dev_chain0:OK #339 tc_links_dev_cleanup:OK #340 tc_links_dev_mixed:OK #341 tc_links_ingress:OK #342 tc_links_invalid:OK #343 tc_links_prepend:OK #344 tc_links_replace:OK #345 tc_links_revision:OK Summary: 14/0 PASSED, 0 SKIPPED, 0 FAILED Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Cc: Martin KaFai Lau <martin.lau@kernel.org> Link: https://lore.kernel.org/r/20240708133130.11609-2-daniel@iogearbox.net Signed-off-by: Martin KaFai Lau <martin.lau@kernel.org>
Add three tests for struct_ops using private stack. ./test_progs -t struct_ops_private_stack #336/1 struct_ops_private_stack/private_stack:OK #336/2 struct_ops_private_stack/private_stack_fail:OK #336/3 struct_ops_private_stack/private_stack_recur:OK #336 struct_ops_private_stack:OK The following is a snippet of a struct_ops check_member() implementation: u32 moff = __btf_member_bit_offset(t, member) / 8; switch (moff) { case offsetof(struct bpf_testmod_ops3, test_1): prog->aux->priv_stack_requested = true; prog->aux->recursion_detected = test_1_recursion_detected; fallthrough; default: break; } return 0; The first test is with nested two different callback functions where the first prog has more than 512 byte stack size (including subprogs) with private stack enabled. The second test is a negative test where the second prog has more than 512 byte stack size without private stack enabled. The third test is the same callback function recursing itself. At run time, the jit trampoline recursion check kicks in to prevent the recursion. The recursion_detected() callback function is implemented by the bpf_testmod, the following message in dmesg bpf_testmod: oh no, recursing into test_1, recursion_misses 1 demonstrates the callback function is indeed triggered when recursion miss happens. Signed-off-by: Yonghong Song <yonghong.song@linux.dev> Link: https://lore.kernel.org/r/20241112163938.2225528-1-yonghong.song@linux.dev Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Hello,
I'm currently interested in the packet scheduler using BPF and have done some experiments. However, I've run into some issues when using
bpf_rrandbpf_redscheduler. There are three possibilities:Problem 1 usually occurs, sometimes problem 2 or 3.
However, using
bpf_firstscheduler doesn't cause any problems above.Setup
My experiment setup is as follows:
Client
Server
We tried to understand this problem more deeply through the BPF tracing tool.
Through
stackcount, we found the following question:sk_reset_timeris being called repeatedly.We found this question very similar to the one mentioned in #52:
I was wondering if this was the same issue or I made some mistakes?
I'm really open to disclose more information about the issue.
Thank you in advance.
The text was updated successfully, but these errors were encountered: