mgmtd failed to startup when using namespaces

[rhermsen]

Description

Using FRR 9.1
Host: TinyCoreLinux 14.0

Trying to start FRR when using namespaces takes a long time and mgmtd fails to start.

With FRR 8.2.2 I was able start frr using namespaces.
Host: TinyCoreLinux 13.0

initial state

$ sudo frrinit.sh status ns-frr
Status of watchfrr: FAILED
Status of zebra: FAILED
Status of mgmtd: FAILED
Status of isisd: FAILED
Status of staticd: FAILED

startup
sudo frrinit.sh start ns-frr
This command takes about 55 seconds to finish with the following output.

Starting watchfrr with command: '  /usr/local/sbin/watchfrr -N ns-frr -d  -F traditional  --netns zebra mgmtd isisd staticd'
Started watchfrr

after startup

$ sudo frrinit.sh status ns-frr
Status of watchfrr: running
Status of zebra: running
Status of mgmtd: FAILED
Status of isisd: running
Status of staticd: running

This working fine without use of namespaces:
initial state

$ sudo frrinit.sh status
Status of watchfrr: FAILED
Status of zebra: FAILED
Status of mgmtd: FAILED
Status of isisd: FAILED
Status of staticd: FAILED

startup
$ sudo frrinit.sh start

after startup

$ sudo frrinit.sh status
Status of watchfrr: running
Status of zebra: running
Status of mgmtd: running
Status of isisd: running
Status of staticd: running

Btw, why is status after stopping showing 'FAILED' i.s.o. 'stopped'?

Version

# show version
FRRouting 9.1 (testapps) on Linux(6.1.2-tinycore).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
configured with:
    '--prefix=/usr/local' '--sysconfdir=/usr/local/etc/frr' '--localstatedir=/var/run/frr' '--runstatedir=/var/run/frr' '--bindir=/usr/local/bin' '--sbindir=/usr/local/sbin' '--enable-user=tc' '--enable-group=staff' '--enable-vty-group=staff' '--enable-ldpd' '--enable-pimd' '--enable-multipath=2' '--enable-rtadv'

How to reproduce

Steps to reproduce:
(some steps are TCL specific like tce-load which just installs a package)

At bootup:

[ ! -e "$TCEDIR"/util-linux.tcz ] && sudo -u tc tce-load -wi util-linux.tcz
[ ! -f /usr/local/tce.installed/util-linux ] && sudo -u tc tce-load -i /mnt/vda1/tce/optional/util-linux.tcz
[ ! -e "$TCEDIR"/iproute2.tcz ] && sudo -u tc tce-load -wi iproute2.tcz
[ ! -f /usr/local/tce.installed/iproute2 ] && sudo -u tc tce-load -i /mnt/vda1/tce/optional/iproute2.tcz
ip netns add ns-frr
ip link set dev eth1 netns ns-frr
ip link set dev eth2 netns ns-frr
ip netns exec ns-frr ip link set eth1 up
ip netns exec ns-frr ip link set eth2 up
# below command will only work if /usr/local/etc/frr exist with correct permissions.
[ ! -d "/usr/local/etc/frr/ns-frr" ] && sudo -u tc mkdir -p  /usr/local/etc/frr/ns-frr/

reboot and verify namespaces are configured

Expectations, only interfaces that are part of the specified namespace visible.

alias ns-frr="sudo ip netns exec ns-frr"
ns-frr ip address

$ ns-frr ip address
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
5: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:50:00:00:01:01 brd ff:ff:ff:ff:ff:ff
6: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:50:00:00:01:02 brd ff:ff:ff:ff:ff:ff

ip address

$ ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 1e:81:de:94:7b:5a brd ff:ff:ff:ff:ff:ff
3: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:50:00:00:01:00 brd ff:ff:ff:ff:ff:ff
    inet 192.168.145.134/24 brd 192.168.145.255 scope global eth0
       valid_lft forever preferred_lft forever
7: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:50:00:00:01:03 brd ff:ff:ff:ff:ff:ff

Configure and start frr

alias ns-frr="sudo ip netns exec ns-frr"
cp /usr/local/etc/frr/frr.conf /usr/local/etc/frr/ns-frr/
cp /usr/local/etc/frr/daemons /usr/local/etc/frr/ns-frr/
cp /usr/local/etc/frr/vtysh.conf /usr/local/etc/frr/ns-frr/
echo usr/local/etc/frr/ns-frr/frr.conf >> /opt/.filetool.lst
echo usr/local/etc/frr/ns-frr/daemons >> /opt/.filetool.lst
echo usr/local/etc/frr/ns-frr/vtysh.conf >> /opt/.filetool.lst
sed -i 's/#watchfrr_options=""/watchfrr_options="--netns"/g' /usr/local/etc/frr/ns-frr/daemons
# next line is already included
# echo service integrated-vtysh-config >>  /usr/local/etc/frr/ns-frr/vtysh.conf
sudo frrinit.sh start ns-frr
# this previous step takes very long. There are no error messages.
sudo frrinit.sh status ns-frr
# the previous step shows mgmtd as failed.
vtysh -N ns-frr

I'm able to login. But e.g. cannot save the config.

# write file
Note: this version of vtysh never writes vtysh.conf
% Not all daemons are up, cannot write config.

Expected behavior

status of mgmtd: running

Actual behavior

Trying to start frr takes a long time (e.g. 55 seconds) and results in Status of mgmtd: FAILED in the sudo frrinit.sh status output.

Additional context

No response

Checklist

• I have searched the open issues for this bug.
• I have not included sensitive information in this report.

[knightsyt]

please check you runstatedir /var/run/frr ; chown -R frr /var/run/frr

[rhermsen]

@knightsyt
Thanks for your update. That is indeed working ( sudo chown -R tc:staff /var/run/frr in my case ).

Did you run into the same issue?

Allowing access to .pid files for none root users is a security risk.
So be careful doing this.

[github-actions]

This issue is stale because it has been open 180 days with no activity. Comment or remove the autoclose label in order to avoid having this issue closed.

[frrbot]

This issue will be automatically closed in the specified period unless there is further activity.