Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] Crash when deleting an online node #2118

Closed
2 of 4 tasks
nblock opened this issue Sep 9, 2024 · 0 comments · Fixed by #2125
Closed
2 of 4 tasks

[Bug] Crash when deleting an online node #2118

nblock opened this issue Sep 9, 2024 · 0 comments · Fixed by #2125
Labels
bug Something isn't working
Milestone

Comments

@nblock
Copy link
Contributor

nblock commented Sep 9, 2024

Is this a support request?

  • This is not a support request

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

A panic occurs when an online node is deleted:

2024-09-09T19:50:58+02:00 INF github.com/juanfont/headscale/hscontrol/auth.go:28 > Successfully sent auth url: http://headscale.lan:8080/register/mkey:622647d93d72d3edf2bff8d179495dd99243a8a75e631250b17d62a40029b712 expiry=-62135596800 fo
llowup=http://headscale.lan:8080/register/mkey:622647d93d72d3edf2bff8d179495dd99243a8a75e631250b17d62a40029b712 machine_key=[YiZH2] node=DESKTOP-T8CLBV9 node_key=[8WSlh] node_key_old=
2024-09-09T19:50:58+02:00 DBG New noise client cap_ver=104 handler=/key  
2024-09-09T19:50:58+02:00 DBG github.com/juanfont/headscale/hscontrol/auth.go:596 > Client is registered and we have the current NodeKey. All clear to /map node=DESKTOP-T8CLBV9
2024-09-09T19:50:58+02:00 INF github.com/juanfont/headscale/hscontrol/auth.go:627 > Node successfully authorized node=DESKTOP-T8CLBV9
2024-09-09T19:50:58+02:00 INF github.com/juanfont/headscale/hscontrol/poll.go:698 > node has connected, mapSession: 0xc0001a2480, chan: 0xc0001fb810 node=DESKTOP-T8CLBV9 node.id=3 omitPeers=false readOnly=false stream=true
2024-09-09T19:51:48+02:00 ERR github.com/juanfont/headscale/hscontrol/poll.go:718 > Could not get machine from db error="record not found" node=DESKTOP-T8CLBV9 node.id=3 omitPeers=false readOnly=false stream=true
2024/09/09 19:51:48 http2: panic serving 192.168.191.238:54790: runtime error: invalid memory address or nil pointer dereference
goroutine 2050 [running]:                                                                                              
golang.org/x/net/http2.(*serverConn).runHandler.func1()                                                                
        golang.org/x/net@v0.28.0/http2/server.go:2407 +0x145
panic({0x1dadb00?, 0x34d2a30?})                      
        runtime/panic.go:785 +0x132                      
github.com/juanfont/headscale/hscontrol.(*mapSession).serveLongPoll.func1()
        github.com/juanfont/headscale/hscontrol/poll.go:206 +0x8a
github.com/juanfont/headscale/hscontrol.(*mapSession).serveLongPoll(0xc0001a2480)
        github.com/juanfont/headscale/hscontrol/poll.go:290 +0x11f4
github.com/juanfont/headscale/hscontrol.(*noiseServer).NoisePollNetMapHandler(0xc000589290, {0x2464940, 0xc00038ca80}, 0xc000266780)
        github.com/juanfont/headscale/hscontrol/noise.go:240 +0x365
net/http.HandlerFunc.ServeHTTP(0xaad78?, {0x2464940?, 0xc00038ca80?}, 0xc000290cf0?)
        net/http/server.go:2220 +0x29
github.com/juanfont/headscale/hscontrol.prometheusMiddleware.func1({0x2464940, 0xc00038ca80}, 0xc000266780)
        github.com/juanfont/headscale/hscontrol/metrics.go:87 +0x143
net/http.HandlerFunc.ServeHTTP(0xc000266640?, {0x2464940?, 0xc00038ca80?}, 0x7fb68785b5b8?)
        net/http/server.go:2220 +0x29
github.com/gorilla/mux.(*Router).ServeHTTP(0xc0006120c0, {0x2464940, 0xc00038ca80}, 0xc000266500)
        github.com/gorilla/mux@v1.8.1/mux.go:212 +0x1e2
golang.org/x/net/http2.(*serverConn).runHandler(0x44951b?, 0xc00052bae0?, 0x24674f0?, 0xc00052bdb0?)
        golang.org/x/net@v0.28.0/http2/server.go:2414 +0xf5 
created by golang.org/x/net/http2.(*serverConn).scheduleHandler in goroutine 1960
        golang.org/x/net@v0.28.0/http2/server.go:2348 +0x21d
2024-09-09T19:51:48+02:00 ERR Failed to fetch node from the database with node key: nodekey:f164a587d1b77539def9790a4085e950556f231d25e9f8dec125c121d7d6c46c handler=NoisePollNetMap
2024-09-09T19:51:48+02:00 ERR Failed to fetch node from the database with node key: nodekey:f164a587d1b77539def9790a4085e950556f231d25e9f8dec125c121d7d6c46c handler=NoisePollNetMap
2024-09-09T19:51:48+02:00 ERR Failed to fetch node from the database with node key: nodekey:f164a587d1b77539def9790a4085e950556f231d25e9f8dec125c121d7d6c46c handler=NoisePollNetMap

Expected Behavior

The node should be removed and headscale should not crash.

Steps To Reproduce

  • Connect a new node with headscale; headscale nodes list shows the as "online"
  • Delete the node: headscale nodes delete -i ID

Environment

- OS: Debian 12
- Headscale version: v0.23.0-beta.4 (but also seen earlier today with beta3)
- Tailscale version: 1.72.0

Runtime environment

  • Headscale is behind a (reverse) proxy
  • Headscale runs in a container

Anything else?

No response

@nblock nblock added the bug Something isn't working label Sep 9, 2024
@kradalby kradalby added this to the v0.23.0 milestone Sep 10, 2024
kradalby added a commit to kradalby/headscale that referenced this issue Sep 11, 2024
Currently we will read the node from database, and since it is
deleted, the id might be set to nil. Keep the node around and
just shutdown, so it is cleanly removed from notifier.

Fixes juanfont#2118

Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
kradalby added a commit to kradalby/headscale that referenced this issue Sep 11, 2024
Currently we will read the node from database, and since it is
deleted, the id might be set to nil. Keep the node around and
just shutdown, so it is cleanly removed from notifier.

Fixes juanfont#2118

Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
kradalby added a commit to kradalby/headscale that referenced this issue Sep 30, 2024
* add shutdown that asserts if headscale had panics

Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>

* add test case producing 2118 panic

Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>

* make stream shutdown if self-node has been removed

Currently we will read the node from database, and since it is
deleted, the id might be set to nil. Keep the node around and
just shutdown, so it is cleanly removed from notifier.

Fixes juanfont#2118

Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>

---------

Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants