alpha9 takes 90 seconds to stop

[mooncaptain108]

Bug description

After upgrading from alpha8 to alpha9 I found that the stop increased from almost instantly to 90 seconds.
systemctl stop headscale
here's the journalctl log

Apr 22 12:05:09 headscale systemd[1]: Stopping headscale coordination server for Tailscale...
Apr 22 12:05:09 headscale headscale[4098]: 2024-04-22T12:05:09-05:00 INF Received signal to stop, shutting down gracefully signal=terminated
Apr 22 12:06:39 headscale systemd[1]: headscale.service: State 'stop-sigterm' timed out. Killing.
Apr 22 12:06:39 headscale systemd[1]: headscale.service: Killing process 4098 (headscale) with signal SIGKILL.
Apr 22 12:06:39 headscale systemd[1]: headscale.service: Killing process 4099 (headscale) with signal SIGKILL.
Apr 22 12:06:39 headscale systemd[1]: headscale.service: Killing process 4100 (headscale) with signal SIGKILL.
Apr 22 12:06:39 headscale systemd[1]: headscale.service: Killing process 4101 (headscale) with signal SIGKILL.
Apr 22 12:06:39 headscale systemd[1]: headscale.service: Killing process 4102 (headscale) with signal SIGKILL.
Apr 22 12:06:39 headscale systemd[1]: headscale.service: Killing process 4103 (headscale) with signal SIGKILL.
Apr 22 12:06:39 headscale systemd[1]: headscale.service: Killing process 4104 (headscale) with signal SIGKILL.
Apr 22 12:06:39 headscale systemd[1]: headscale.service: Killing process 4105 (headscale) with signal SIGKILL.
Apr 22 12:06:39 headscale systemd[1]: headscale.service: Main process exited, code=killed, status=9/KILL
Apr 22 12:06:39 headscale systemd[1]: headscale.service: Failed with result 'timeout'.
Apr 22 12:06:39 headscale systemd[1]: Stopped headscale coordination server for Tailscale.
Apr 22 12:06:39 headscale systemd[1]: headscale.service: Consumed 33.940s CPU time.

Environment

• OS: ubuntu 22.04
• Headscale version: v0.23.0-alpha9
• Tailscale version: 1.64.0

• [x ] Headscale is behind a (reverse) proxy
• Headscale runs in a container

To Reproduce

0. a running alpha8 release - which I could stop and start quickly
1. download the alpha9.deb
2. apt install headscale.deb
3. start headscale
4. I was doing some testing from a remote node and noticed that not all nodes were showing up in my tailscale status list.
5. made some changes to the acl and restarted headscale.
6. Noticed the delay - it finally came back and I started to dig in.
   My tailscale status problem was resolved after the restart.
7. I reverted my acl changes and that didn't make difference.
8. I copied my backup db back in. and that didn't help.
9. I did just: systemctl stop headscale and found that the stop was where the delay occurred
10. described the problem on discord.

Logs and attachments

See above for journalctl output

[cy1der]

+1 on this, same issue

[remisharrock]

same here

[kradalby]

@mooncaptain108, I'm struggling to replicate this in my own setup, could you try #1922 and post the shutdown log?

[mooncaptain108]

Hi Juan, Please forgive my ignorance here. I would really like to help. It looks like #1922 is a code change that would require me to build headscale locally. Or is there a build asset (amd64.deb) that I can download and install? I could probably figure out how to do the build. I have dabbled a tiny bit in go but I may not get it right. Cheers, paul

…

________________________________ From: Kristoffer Dalby ***@***.***> Sent: Thursday, May 2, 2024 4:32 AM To: juanfont/headscale ***@***.***> Cc: mooncaptain108 ***@***.***>; Mention ***@***.***> Subject: Re: [juanfont/headscale] alpha9 takes 90 seconds to stop (Issue #1911) @mooncaptain108<https://github.com/mooncaptain108>, I'm struggling to replicate this in my own setup, could you try #1922<#1922> and post the shutdown log? — Reply to this email directly, view it on GitHub<#1911 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AQZHACBBV5L5LH47LQWT273ZAIBZ7AVCNFSM6AAAAABGTHHGYGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOJQGAYTIMJQGM>. You are receiving this because you were mentioned.

[kradalby]

I just released https://github.com/juanfont/headscale/releases/tag/v0.23.0-alpha11, the change is in there.

[mooncaptain108]

The problem is not present in alpha11.
I did a systemctl stop headscale
and here's what I got in journalctl.

May 02 09:44:43 headscale systemd[1]: Stopping headscale coordination server for Tailscale...
May 02 09:44:43 headscale headscale[5935]: 2024-05-02T09:44:43-05:00 INF Received signal to stop, shutting down gracefully signal=terminated
May 02 09:44:43 headscale headscale[5935]: 2024-05-02T09:44:43-05:00 INF Headscale stopped
May 02 09:44:43 headscale headscale[5935]: 2024-05-02T09:44:43-05:00 FTL ../../../home/runner/work/headscale/headscale/cmd/headscale/cli/server.go:26 > Error starting server error="http: Server closed"
May 02 09:44:43 headscale systemd[1]: headscale.service: Main process exited, code=exited, status=1/FAILURE
May 02 09:44:43 headscale systemd[1]: headscale.service: Failed with result 'exit-code'.
May 02 09:44:43 headscale systemd[1]: Stopped headscale coordination server for Tailscale.

headscale stopped immediately and I was able also to do restart and have it immediately startup again.

Just so you know when I installed alpha11 the install script had to wait 90 seconds to shutdown alpha10.

[mooncaptain108]

this problem is closed for me.