LDPd crash in RB Tree rbe_insert_color()

[mwinter-osr]

Debug information is from master branch at dceb5f8 (July 20). However the bug exists at least since f30a732 (June 26), but not on or before bade23d (June 2)
Bug does not exist on 2.0 or 3.0

LDPd crashes in ANVL Compliance Test ANVL-LDP-28.12

From the LDP log:

2017/07/20 18:04:01 LDP: msg[out]: initialization: lsr-id 192.168.1.1
2017/07/20 18:04:01 LDP: kalive[out]: keepalive: lsr-id 192.168.1.1
2017/07/20 18:04:01 LDP: kalive[in]: keepalive: lsr-id 192.168.1.1
2017/07/20 18:04:01 LDP: nbr_fsm: event KEEPALIVE RECEIVED resulted in action START NEIGHBOR SESSION and changing state for lsr-id 192.168.1.1 from OPENREC to OPERATIONAL
2017/07/20 18:04:02 LDP: waiting for children to terminate
2017/07/20 18:04:02 LDP: terminating

Backtrace info from the coredump:

#0  0x00007fde6d308428 in __GI_raise (sig=sig@entry=6)
	at ../sysdeps/unix/sysv/linux/raise.c:54
#1  0x00007fde6d30a02a in __GI_abort () at abort.c:89
#2  0x00007fde6db1397b in core_handler (signo=11, siginfo=0x7ffdab726170, 
	context=0x7ffdab726040) at sigevent.c:249
#3  <signal handler called>
#4  rbe_insert_color (rbe=0x113daa0, rbt=<optimized out>, t=<optimized out>)
	at openbsd-tree.c:169
#5  _rb_insert (t=t@entry=0x42b1d0 <nbr_pid_head_RB_INFO>, 
	rbt=rbt@entry=0x63d220 <nbrs_by_pid>, elm=elm@entry=0x113da60)
	at openbsd-tree.c:432
#6  0x0000000000417153 in nbr_pid_head_RB_INSERT (elm=0x113da60, 
	head=0x63d220 <nbrs_by_pid>) at ldpe.h:124
#7  nbr_update_peerid (nbr=nbr@entry=0x113da60) at neighbor.c:338
#8  0x0000000000417196 in nbr_act_session_operational (nbr=nbr@entry=0x113da60)
	at neighbor.c:750
#9  0x00000000004174f1 in nbr_fsm (nbr=nbr@entry=0x113da60, 
	event=event@entry=NBR_EVT_KEEPALIVE_RCVD) at neighbor.c:182
#10 0x00000000004272eb in recv_keepalive (nbr=nbr@entry=0x113da60, 
	buf=buf@entry=0x113d3ea "\002\001", len=len@entry=8) at keepalive.c:60
#11 0x0000000000419b63 in session_read (thread=<optimized out>) at packet.c:546
#12 0x00007fde6daf5ef7 in thread_call (thread=thread@entry=0x7ffdab726970)
	at thread.c:1470
#13 0x00000000004140f5 in ldpe () at ldpe.c:136
#14 0x000000000040b4eb in main (argc=0, argv=<optimized out>) at ldpd.c:280

#4 is here:

#4  rbe_insert_color (rbe=0x113daa0, rbt=<optimized out>, t=<optimized out>)
	at openbsd-tree.c:169
169			if (parent == RBE_LEFT(gparent)) {
(gdb) list
164	
165		while ((parent = RBE_PARENT(rbe)) != NULL
166		       && RBE_COLOR(parent) == RB_RED) {
167			gparent = RBE_PARENT(parent);
168	
169			if (parent == RBE_LEFT(gparent)) {
170				tmp = RBE_RIGHT(gparent);
171				if (tmp != NULL && RBE_COLOR(tmp) == RB_RED) {
172					RBE_COLOR(tmp) = RB_BLACK;
173					rbe_set_blackred(parent, gparent);

and it crashes as gparent is NULL in line 169

parent is {rbt_parent = 0x0, rbt_left = 0x0, rbt_right = 0x113daa0, rbt_color = 1}

Issue is reliable reproducable. Access info of the bug is mailed to @rwestphal to look at.

[dwalton76]

I am able to reproduce this on one of our internal tests (tests.nclu.bond_and_vrr_test.VrrNcluTest.test06_bond_lacp_bypass) as well

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/lib/frr/zebra -M snmp -s 90000000 --daemon -A 127.0.0.1'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f1e882ac067 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#0  0x00007f1e882ac067 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007f1e882ad448 in __GI_abort () at abort.c:89
#2  0x00007f1e895f8c8f in core_handler (signo=11, siginfo=0x7ffcbc660e70, context=0x7ffcbc660d40) at sigevent.c:249
#3  <signal handler called>
#4  rbe_insert_color (rbe=0x1a517f0, rbt=<optimized out>, t=<optimized out>) at openbsd-tree.c:169
#5  _rb_insert (t=t@entry=0x7f1e8982b430 <vrf_id_head_RB_INFO>, rbt=0x6615c0 <vrfs_by_id>, rbt@entry=0x7f1e89834dc0 <vrfs_by_id>, elm=elm@entry=0x1a517f0) at openbsd-tree.c:432
#6  0x00007f1e895fc36c in vrf_id_head_RB_INSERT (head=0x7f1e89834dc0 <vrfs_by_id>, elm=0x1a517f0) at vrf.h:92
#7  vrf_get (vrf_id=<optimized out>, name=name@entry=0x7ffcbc661674 "vrf1") at vrf.c:124
#8  0x00000000004355e2 in netlink_vrf_change (h=h@entry=0x7ffcbc661650, tb=<optimized out>, name=name@entry=0x7ffcbc661674 "vrf1") at if_netlink.c:343
#9  0x00000000004368d1 in netlink_link_change (snl=<optimized out>, h=0x7ffcbc661650, ns_id=<optimized out>, startup=<optimized out>) at if_netlink.c:1083
#10 0x000000000043ad13 in netlink_parse_info (filter=filter@entry=0x43a457 <netlink_information_fetch>, nl=nl@entry=0x19412e8, zns=zns@entry=0x19412c0, count=count@entry=5, startup=startup@entry=0) at kernel_netlink.c:636
#11 0x000000000043ada0 in kernel_read (thread=<optimized out>) at kernel_netlink.c:273
#12 0x00007f1e895e0789 in thread_call (thread=thread@entry=0x7ffcbc663870) at thread.c:1470
#13 0x00007f1e896019ca in frr_run (master=0x18a81b0) at libfrr.c:435
#14 0x000000000040d754 in main (argc=8, argv=0x7ffcbc663a48) at main.c:344

[rwestphal]

@dwalton76 could you check if #848 fixes the problem for you?

[dwalton76]

@rwestphal yep this fixes the crash for me