Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BasicHost submits EvtLocalAddressesUpdated but addresses were not changed #2972

Closed
Wondertan opened this issue Sep 21, 2024 · 4 comments · Fixed by #2980
Closed

BasicHost submits EvtLocalAddressesUpdated but addresses were not changed #2972

Wondertan opened this issue Sep 21, 2024 · 4 comments · Fixed by #2980
Labels
kind/bug A bug in existing code (including security flaws)

Comments

@Wondertan
Copy link
Contributor

Context

I am running a test network in the k8s cluster, and while debugging an orthogonal issue, I observed unexpected behavior.

Bug (?)

To my knowledge, the addresses of running pods do not change. Also, the logs below show that the addresses on every event are constant, however BasicHost still submits the EvtLocalAddressesUpdated .

Trace

The trace was from PubSub node spamming go-libp2p-pubsub@v0.12.0/pubsub.go:698 already have connection to peer, triggered by EvtPeerIdentificationCompleted, initiated by identify push from a peer, prompted by BasicHost background loop on the peer detecting address changes.

However, as per identify logs on the PubSub node, the addresses didn't change, and the exact same log with the same address repeats over and over, potentially suggesting that diffing logic in makeUpdatedAddrsEvent has a bug.

DEBUG net/identify identify/id.go:832 received listen addrs for 12D3KooWQHiNsPQoiRMnixa1SzEMUukJo3r1UQ7tZaxH8kjMSm7L: [/ip4/151.115.12.199/udp/2121/quic-v1 /ip4/151.115.12.199/tcp/2121 /ip4/100.64.42.71/tcp/2121 /ip4/100.64.42.71/udp/2121/quic-v1 /ip4/100.64.42.71/udp/2121/quic-v1/webtransport/certhash/uEiCRTETNTiCP0LDvirFICa3zihf3qB14vOa6F7N-GoFntQ/certhash/uEiAr9jX-WLX19pVfR5MY1vL2y_ytxxcRQNPtCfE6oVlH_Q /ip4/100.64.42.71/udp/2121/quic-v1/webtransport/certhash/uEiCRTETNTiCP0LDvirFICa3zihf3qB14vOa6F7N-GoFntQ/certhash/uEiAr9jX-WLX19pVfR5MY1vL2y_ytxxcRQNPtCfE6oVlH_Q]

I couldn't find a way to inspect the state of the peer submitting the identify push with logs, and the only data point is the log from the peer above.

@sukunrt
Copy link
Member

sukunrt commented Sep 23, 2024

I'm unable to reproduce this. Is it possible that the id pushes were triggered by Local Protocols Updated event?

@sukunrt sukunrt added the kind/bug A bug in existing code (including security flaws) label Sep 23, 2024
@Wondertan
Copy link
Contributor Author

Hello @sukunrt! That would suggest we have a constant loop that registers and unregisters some protocol(s), which might be a bug on our side. From the top of your head, do you know how to check this?

@Wondertan
Copy link
Contributor Author

Ok, metrics should help here

@Wondertan
Copy link
Contributor Author

Wondertan commented Sep 24, 2024

Ok, this confirms it's caused by EvtLocalAddressesUpdated via the addresses_updated chart. You can also see the rate at which these messages are sent, which equals the background loop in BasicHost
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug A bug in existing code (including security flaws)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants