-
Notifications
You must be signed in to change notification settings - Fork 228
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
go-libp2p-kad-dht version 2 #864
Closed
Closed
Changes from 4 commits
Commits
Show all changes
65 commits
Select commit
Hold shift + click to select a range
7d18e89
Start go-libp2p-kad-dht v2 package
dennis-tra 5bec109
improve DHT struct field documentation
dennis-tra 311f0d3
add: default bootstrap peers method
dennis-tra dfe3a81
go mod tidy
dennis-tra 57c4988
use slog for logging
dennis-tra 984f28e
handle reachability change events
dennis-tra 35a977f
Add network event subscription
dennis-tra e8e5be2
Add stream handler implementation
dennis-tra 027c8d2
Add FIND_NODE and PING handlers
dennis-tra 3171894
add handlers and tests
dennis-tra 9f8f4a7
mark triage and gokad parts
dennis-tra 3cdfb34
test put value handler
dennis-tra c584b02
add get value tests
dennis-tra 4dfcd8b
remove unused methods
dennis-tra 0af762c
update handlers
dennis-tra 741843b
introduce backend concept
dennis-tra 5e09e95
improve backend documentation
dennis-tra 619429c
introduce protocol constants
dennis-tra 3648184
simplify key handling
dennis-tra 015617e
improve documentation
dennis-tra 38c2d32
add add providers tests
dennis-tra 7a0e26c
improve telemetry
dennis-tra a8a222b
Add AddressFilter feature
dennis-tra 6bdd7ec
WIP
dennis-tra 14c478c
test backend provider garbage collection
dennis-tra 2643b72
refactor provider backend tests
dennis-tra 890d4ec
add stream tests
dennis-tra bdac291
add logErr helper method
dennis-tra 50f8d00
babysteps towards routing.Routing and kademlia.Router
dennis-tra 5a30d89
update deps
dennis-tra 4660bbc
add test for unregistered message type
dennis-tra 9f88f20
add comments
dennis-tra 9559064
clean up backends in DHT close
dennis-tra 4418cfb
add notifee test
dennis-tra 1cbb419
moved two handler tests from v1
dennis-tra 57c5df3
Move kademlia types to own package
dennis-tra 99dbdc6
Add backend tracing
dennis-tra f16d79e
improve kadt package doc
dennis-tra 2610f89
document tracedBackend
dennis-tra 54f20b5
Integrate Zikade/go-kademlia in v2 (#880)
dennis-tra 722e958
Simplify usage of kadtest.CtxShort
iand add1d60
Merge pull request #885 from libp2p/v2-ctx-short
iand 7d838c5
Give dht and coordinator their own telemetry instances (#891)
iand 77dbff0
Migrate go-kademlia state machines (#893)
iand 97e4e02
v2: upgrade build to include go1.21 (#890)
iand 1d35505
Test query interactions with routing table (#887)
iand 3723b8a
remove jaeger dependency (#900)
dennis-tra 156aab2
fix: avoid panic when node is re-added to probe list (#902)
dennis-tra 1220ddd
Decouple coord package from addressing (#903)
iand e86381e
refactor: v2 simplify tracing (#924)
dennis-tra 83329a4
Clean up DHT test helpers (#928)
iand 2da54ab
Improve query capabilities (#932)
iand 74ffa67
Add broadcast state machine for storing records in the DHT (#930)
dennis-tra ae5a094
Add explore state machine to expand population of routing table (#934)
iand 09dd7b0
Expose behaviour and state machine configs (#937)
iand dd5e537
feat: findProvidersAsync (#938)
dennis-tra dedca86
Add metrics to routing state machines (#939)
iand e4b1034
add provide routing tests (#940)
dennis-tra 4fa560f
fix: bootstrap state machine goes idle after completion (#943)
iand 6a4249c
Logging improvements (#941)
iand 03adce6
Implement SearchValue/GetValue (#942)
dennis-tra 90d748b
fix: missing QueryMessage return value
dennis-tra 0e628c0
fix: use correct err log method
dennis-tra 509eee4
fix: avoid panic in bootstrap when late messages arrive (#949)
iand 1d1fe93
Use go-libdht (#952)
iand File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,34 @@ | ||
package dht | ||
|
||
import ( | ||
"github.com/libp2p/go-libp2p/core/peer" | ||
) | ||
|
||
// defaultBootstrapPeers is a set of hard-coded public DHT bootstrap peers | ||
// operated by Protocol Labs. This slice is filled in the init() method. | ||
var defaultBootstrapPeers []peer.AddrInfo | ||
|
||
func init() { | ||
for _, s := range []string{ | ||
"/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN", | ||
"/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa", | ||
"/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb", | ||
"/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt", | ||
"/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ", // mars.i.ipfs.io | ||
} { | ||
addrInfo, err := peer.AddrInfoFromString(s) | ||
if err != nil { | ||
panic(err) | ||
} | ||
defaultBootstrapPeers = append(defaultBootstrapPeers, *addrInfo) | ||
} | ||
} | ||
|
||
// DefaultBootstrapPeers returns hard-coded public DHT bootstrap peers operated | ||
// by Protocol Labs. You can configure your own set of bootstrap peers by | ||
// overwriting the corresponding Config field. | ||
func DefaultBootstrapPeers() []peer.AddrInfo { | ||
peers := make([]peer.AddrInfo, len(defaultBootstrapPeers)) | ||
copy(peers, defaultBootstrapPeers) | ||
return peers | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,23 @@ | ||
package dht | ||
|
||
import ( | ||
"testing" | ||
|
||
"github.com/libp2p/go-libp2p/core/peer" | ||
) | ||
|
||
func TestDefaultBootstrapPeers(t *testing.T) { | ||
bps := DefaultBootstrapPeers() | ||
if len(bps) != len(defaultBootstrapPeers) { | ||
t.Errorf("len(DefaultBootstrapPeers()) = %d, want %v", len(bps), len(defaultBootstrapPeers)) | ||
} | ||
|
||
bpmap := make(map[peer.ID]peer.AddrInfo) | ||
for _, info := range bps { | ||
bpmap[info.ID] = info | ||
} | ||
|
||
if len(bpmap) != len(defaultBootstrapPeers) { | ||
t.Errorf("unique DefaultBootstrapPeers() = %d, want %v", len(bpmap), len(defaultBootstrapPeers)) | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,105 @@ | ||
package dht | ||
|
||
import ( | ||
"fmt" | ||
|
||
"github.com/plprobelab/go-kademlia/coord" | ||
"github.com/plprobelab/go-kademlia/kad" | ||
"github.com/plprobelab/go-kademlia/key" | ||
) | ||
|
||
type ( | ||
// ModeOpt describes in which mode this DHT process should operate in. | ||
// Possible options are client, server, and any variant that switches | ||
// between both automatically based on public reachability. The DHT receives | ||
// reachability updates from libp2p via the EvtLocalReachabilityChanged | ||
// event. A DHT that operates in client mode won't register a stream handler | ||
// for incoming requests and therefore won't store, e.g., any provider or | ||
// IPNS records. A DHT in server mode, on the other hand, does all of that. | ||
// | ||
// The `mode` type, on the other hand, captures the current state that the | ||
// DHT is in. This can either be client or server. | ||
ModeOpt string | ||
|
||
// mode describes in which mode the DHT currently operates. Because the ModeOpt | ||
// type has options that automatically switch between client and server mode | ||
// based on public connectivity, the DHT mode at any point in time can differ | ||
// from the desired mode. Therefore, we define this second mode type that | ||
// only has the two forms: client or server. | ||
mode string | ||
) | ||
|
||
const ( | ||
// ModeOptClient configures the DHT to only operate in client mode | ||
// regardless of potential public reachability. | ||
ModeOptClient ModeOpt = "client" | ||
|
||
// ModeOptServer configures the DHT to always operate in server mode | ||
// regardless of potentially not being publicly reachable. | ||
ModeOptServer ModeOpt = "server" | ||
|
||
// ModeOptAutoClient configures the DHT to start operating in client mode | ||
// and if publicly reachability is detected to switch to server mode. | ||
ModeOptAutoClient ModeOpt = "auto-client" | ||
|
||
// ModeOptAutoServer configures the DHT to start operating in server mode, | ||
// and if it is detected that we don't have public reachability switch | ||
// to client mode. | ||
ModeOptAutoServer ModeOpt = "auto-server" | ||
|
||
// modeClient means that the DHT is currently operating in client mode. | ||
// For more information, check ModeOpt documentation. | ||
modeClient mode = "client" | ||
|
||
// modeServer means that the DHT is currently operating in server mode. | ||
// For more information, check ModeOpt documentation. | ||
modeServer mode = "server" | ||
) | ||
|
||
// Config contains all the configuration options for a DHT. Use DefaultConfig | ||
// to build up your own configuration struct. The DHT constructor New uses the | ||
// below method Validate to test for violations of configuration invariants. | ||
type Config struct { | ||
// Mode defines if the DHT should operate as a server or client or switch | ||
// between both automatically (see ModeOpt). | ||
Mode ModeOpt | ||
|
||
// Kademlia holds the configuration of the underlying Kademlia implementation. | ||
Kademlia *coord.Config | ||
|
||
// RoutingTable holds a reference to the specific routing table | ||
// implementation that this DHT should use. If this field is nil, the | ||
// triert.TrieRT routing table will be used. | ||
RoutingTable kad.RoutingTable[key.Key256, kad.NodeID[key.Key256]] | ||
} | ||
|
||
// DefaultConfig returns a configuration struct that can be used as-is to | ||
// instantiate a fully functional DHT client. | ||
func DefaultConfig() *Config { | ||
return &Config{ | ||
Mode: ModeOptAutoClient, | ||
Kademlia: coord.DefaultConfig(), | ||
RoutingTable: nil, | ||
} | ||
} | ||
|
||
// Validate validates the configuration struct it is called on. It returns | ||
// an error if any configuration issue was detected and nil if this is | ||
// a valid configuration. | ||
func (c *Config) Validate() error { | ||
switch c.Mode { | ||
case ModeOptClient, ModeOptServer, ModeOptAutoClient, ModeOptAutoServer: | ||
default: | ||
return fmt.Errorf("invalid mode option: %s", c.Mode) | ||
} | ||
|
||
if c.Kademlia == nil { | ||
return fmt.Errorf("kademlia configuration must not be nil") | ||
} | ||
|
||
if err := c.Kademlia.Validate(); err != nil { | ||
return fmt.Errorf("invalid kademlia configuration: %w", err) | ||
} | ||
|
||
return nil | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,52 @@ | ||
package dht | ||
|
||
import ( | ||
"testing" | ||
) | ||
|
||
func TestConfig_Validate(t *testing.T) { | ||
tests := []struct { | ||
name string | ||
mutate func(*Config) *Config | ||
wantErr bool | ||
}{ | ||
{ | ||
name: "happy path", | ||
wantErr: false, | ||
mutate: func(c *Config) *Config { return c }, | ||
}, | ||
{ | ||
name: "invalid mode", | ||
wantErr: true, | ||
mutate: func(c *Config) *Config { | ||
c.Mode = "invalid" | ||
return c | ||
}, | ||
}, | ||
{ | ||
name: "nil Kademlia configuration", | ||
wantErr: true, | ||
mutate: func(c *Config) *Config { | ||
c.Kademlia = nil | ||
return c | ||
}, | ||
}, | ||
{ | ||
name: "invalid Kademlia configuration", | ||
wantErr: true, | ||
mutate: func(c *Config) *Config { | ||
c.Kademlia.Clock = nil | ||
return c | ||
}, | ||
}, | ||
} | ||
for _, tt := range tests { | ||
t.Run(tt.name, func(t *testing.T) { | ||
c := DefaultConfig() | ||
c = tt.mutate(c) | ||
if err := c.Validate(); (err != nil) != tt.wantErr { | ||
t.Errorf("Validate() error = %v, wantErr %v", err, tt.wantErr) | ||
} | ||
}) | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,82 @@ | ||
package dht | ||
|
||
import ( | ||
"fmt" | ||
|
||
"github.com/libp2p/go-libp2p/core/host" | ||
ma "github.com/multiformats/go-multiaddr" | ||
"github.com/plprobelab/go-kademlia/coord" | ||
"github.com/plprobelab/go-kademlia/kad" | ||
"github.com/plprobelab/go-kademlia/key" | ||
"github.com/plprobelab/go-kademlia/routing/triert" | ||
) | ||
|
||
// DHT is an implementation of Kademlia with S/Kademlia modifications. | ||
// It is used to implement the base Routing module. | ||
type DHT struct { | ||
// host holds a reference to the underlying libp2p host | ||
host host.Host | ||
|
||
// cfg holds a reference to the DHT configuration struct | ||
cfg *Config | ||
|
||
// mode indicates the current mode the DHT operates in. This can differ from | ||
// the desired mode if set to auto-client or auto-server. The desired mode | ||
// can be configured via the Config struct. | ||
mode mode | ||
|
||
// kad is a reference to the go-kademlia coordinator | ||
kad *coord.Coordinator[key.Key256, ma.Multiaddr] | ||
|
||
// rt holds a reference to the routing table implementation. This can be | ||
// configured via the Config struct. | ||
rt kad.RoutingTable[key.Key256, kad.NodeID[key.Key256]] | ||
} | ||
|
||
// New constructs a new DHT for the given underlying host and with the given | ||
// configuration. Use DefaultConfig() to construct a configuration. | ||
func New(h host.Host, cfg *Config) (*DHT, error) { | ||
var err error | ||
|
||
// check if the configuration is valid | ||
if err = cfg.Validate(); err != nil { | ||
return nil, fmt.Errorf("validate DHT config: %w", err) | ||
} | ||
|
||
d := &DHT{ | ||
host: h, | ||
cfg: cfg, | ||
} | ||
|
||
nid := nodeID(d.host.ID()) | ||
|
||
// Use the configured routing table if it was provided | ||
if cfg.RoutingTable != nil { | ||
d.rt = cfg.RoutingTable | ||
} else { | ||
rtCfg := triert.DefaultConfig[key.Key256, kad.NodeID[key.Key256]]() | ||
d.rt, err = triert.New[key.Key256, kad.NodeID[key.Key256]](nid, rtCfg) | ||
if err != nil { | ||
return nil, fmt.Errorf("new trie routing table: %w", err) | ||
} | ||
} | ||
|
||
// instantiate a new Kademlia DHT coordinator. | ||
d.kad, err = coord.NewCoordinator[key.Key256, ma.Multiaddr](nid, nil, d.rt, cfg.Kademlia) | ||
if err != nil { | ||
return nil, fmt.Errorf("new coordinator: %w", err) | ||
} | ||
|
||
// determine mode to start in | ||
switch cfg.Mode { | ||
case ModeOptClient, ModeOptAutoClient: | ||
d.mode = modeClient | ||
case ModeOptServer, ModeOptAutoServer: | ||
d.mode = modeServer | ||
default: | ||
// should never happen because of the configuration validation above | ||
return nil, fmt.Errorf("invalid dht mode %s", cfg.Mode) | ||
} | ||
|
||
return d, nil | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,78 @@ | ||
package dht | ||
|
||
import ( | ||
"reflect" | ||
"testing" | ||
|
||
"github.com/libp2p/go-libp2p" | ||
) | ||
|
||
func TestNew(t *testing.T) { | ||
h, err := libp2p.New(libp2p.NoListenAddrs) | ||
if err != nil { | ||
t.Fatal(err) | ||
} | ||
|
||
tests := []struct { | ||
name string | ||
cfgBuilder func(*Config) *Config | ||
wantBuilder func(*DHT) *DHT | ||
wantErr bool | ||
}{ | ||
{ | ||
name: "happy path", | ||
cfgBuilder: func(c *Config) *Config { return c }, | ||
wantBuilder: func(dht *DHT) *DHT { return dht }, | ||
wantErr: false, | ||
}, | ||
{ | ||
name: "mode set to server", | ||
cfgBuilder: func(c *Config) *Config { | ||
c.Mode = ModeOptServer | ||
return c | ||
}, | ||
wantBuilder: func(dht *DHT) *DHT { | ||
dht.mode = modeServer | ||
return dht | ||
}, | ||
wantErr: false, | ||
}, | ||
{ | ||
name: "mode set to auto client", | ||
cfgBuilder: func(c *Config) *Config { | ||
c.Mode = ModeOptAutoClient | ||
return c | ||
}, | ||
wantBuilder: func(dht *DHT) *DHT { | ||
dht.mode = modeClient | ||
return dht | ||
}, | ||
wantErr: false, | ||
}, | ||
} | ||
|
||
for _, tt := range tests { | ||
t.Run(tt.name, func(t *testing.T) { | ||
c := DefaultConfig() | ||
d, err := New(h, c) | ||
if err != nil { | ||
t.Fatal(err) | ||
} | ||
|
||
got, err := New(h, tt.cfgBuilder(c)) | ||
if (err != nil) != tt.wantErr { | ||
t.Errorf("New() error = %v, wantErr %v", err, tt.wantErr) | ||
return | ||
} | ||
|
||
want := tt.wantBuilder(d) | ||
|
||
want.kad = nil | ||
got.kad = nil | ||
|
||
if !reflect.DeepEqual(got, want) { | ||
t.Errorf("New() got = %v, want %v", got, want) | ||
} | ||
}) | ||
} | ||
} |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
perhaps we should rename this to
<NETWORK_NAME>BootstrapPeers()
, e.g.,AminoBootstrapPeers