Skip to content
This repository has been archived by the owner on Apr 18, 2024. It is now read-only.

NULL pointer derefence for redundant scheduler #297

Closed
AlexanderFroemmgen opened this issue Nov 14, 2018 · 4 comments
Closed

NULL pointer derefence for redundant scheduler #297

AlexanderFroemmgen opened this issue Nov 14, 2018 · 4 comments
Assignees
Labels

Comments

@AlexanderFroemmgen
Copy link
Contributor

While working on #271, I noticed that the redundant scheduler causes a null pointer dereference in the call to TCP_SKB_CB at https://github.com/multipath-tcp/mptcp/blob/mptcp_trunk/net/mptcp/mptcp_sched.c#L64 with the current mptcp_trunk.

[ 676.788279] BUG: unable to handle kernel NULL pointer dereference at 00000000 00000029
[ 676.788581] mptcp_alloc_mpcb: created mpcb with token 0x2af13eb4
[ 676.796249] PGD 0 P4D 0
[ 676.796253] Oops: 0000 [#1] SMP PTI
[ 676.796255] CPU: 2 PID: 3940 Comm: iperf Not tainted 4.18.0+ #1
[ 676.796256] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006
[ 676.796261] RIP: 0010:mptcp_is_temp_unavailable+0x3d/0x130
[ 676.796263] Code:
[ 676.801838] mptcp_add_sock: token 0x2af13eb4 pi 1, src_addr:10.0.0.1:5001 dst _addr:10.0.0.2:52948
[ 676.804574] 83 e1 3f 80 f9 04 0f 84 a2 00 00 00 4c 8b 87 48 08 00 00 41 0f b6 88 8e 00 00 00 f6 c1 02 75 20 48 85 f6 74 1b 83 e1 04 74 16 <8b> 5e 28 41 39 98 20 01 00 00 b8 01 00 00 00 74 05 5b 41 5c 5d c3
[ 676.844310] RSP: 0018:ffffb062c1f17b28 EFLAGS: 00010202
[ 676.848525] RAX: 0000000000000000 RBX: ffff99a532439a40 RCX: 0000000000000004
[ 676.853948] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff99a532439a40
[ 676.859231] RBP: ffffb062c1f17b38 R08: ffff99a520d7fc80 R09: 0000000000000000
[ 676.864466] R10: 0000000000000130 R11: ffff99a51ef96a58 R12: 0000000000000000
[ 676.869816] R13: 0000000000000001 R14: ffff99a520d7fc80 R15: ffff99a51ef96900
[ 676.874975] FS: 00007f2af321d700(0000) GS:ffff99a57f280000(0000) knlGS:00000 00000000000
[ 676.880637] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 676.885100] CR2: 0000000000000029 CR3: 00000003d11b6001 CR4: 00000000001606e0
[ 676.890345] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 676.895673] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 676.901275] Call Trace:
[ 676.903887] mptcp_is_available+0x31/0x40
[ 676.907320] redsched_use_subflow+0x28/0x70
[ 676.910859] redundant_next_segment+0x28b/0x320

Does anyone have an idea how this problem was introduced? Might this addionally cause problems somewhere else?

What is the semantic of an skb without control block? Is it safe to ignore this packet in the scheduler?

@cpaasch
Copy link
Member

cpaasch commented Nov 14, 2018

There is a check for whether or not skb is != NULL in mptcp_is_temp_unavailable. Thus, this should not be failing.

However, it seems like skb is 0000000000000029. Thus, the NULL-check won't fail. Maybe you are dereferencing garbage-memory in redundant_next_segment ?

@cpaasch
Copy link
Member

cpaasch commented Nov 14, 2018

You could enable CONFIG_KASAN for more explicit debugging info.

@AlexanderFroemmgen
Copy link
Contributor Author

Ok, it took me some time, but I found the problem.

The redundant scheduler uses an own pointer in the skb queue per subflow. Since 75c119a, packets which are in flight are moved out of the skb queue and are stored in an rb tree structure. This breaks the redundant scheduler, as it still tries to follow the skb->next pointer to find the next packet. This fails, as the skb is not part of the queue anymore, but inside the rb tree.

I will try to find a way how to mitigate this problem.

@cpaasch
Copy link
Member

cpaasch commented Dec 7, 2018

Awesome find! Yeah, the rb-tree is causing quite some issues.

cpaasch pushed a commit that referenced this issue Dec 12, 2018
With 75c119a, a new rt-tree based retransmit queue was introduced.
This breaks the assumptions of the redundant scheduler and leads
to failures when the scheduler trys to schedule the next packet b
ased on the skbs in the send queue.

This commit fixes #297

Fixes: c61bc63 ("Merge tag 'v4.15-rc3' into mptcp_trunk")
Signed-off-by: AlexanderFroemmgen <froemmgen@google.com>
Signed-off-by: Christoph Paasch <cpaasch@apple.com>
@cpaasch cpaasch closed this as completed Dec 12, 2018
hoang-tranviet pushed a commit to hoang-tranviet/mptcp_private that referenced this issue Apr 30, 2019
With 75c119a, a new rt-tree based retransmit queue was introduced.
This breaks the assumptions of the redundant scheduler and leads
to failures when the scheduler trys to schedule the next packet b
ased on the skbs in the send queue.

This commit fixes multipath-tcp/mptcp#297

Fixes: c61bc63 ("Merge tag 'v4.15-rc3' into mptcp_trunk")
Signed-off-by: AlexanderFroemmgen <froemmgen@google.com>
Signed-off-by: Christoph Paasch <cpaasch@apple.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

2 participants