Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(routing): Delegated Routing #8997

Merged
merged 27 commits into from
Jul 7, 2022
Merged

feat(routing): Delegated Routing #8997

Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
0638d25
Delegated Routing.
ajnavarro May 26, 2022
104ae3e
Requested changes.
ajnavarro Jun 28, 2022
b742db9
Init using op string
ajnavarro Jun 29, 2022
420c6fd
Separate possible ContentRouters for TopicDiscovery.
ajnavarro Jun 29, 2022
2f93364
Set dht default routing type
ajnavarro Jun 29, 2022
25b9683
Add tests and remove uneeded code
ajnavarro Jun 29, 2022
0f549a2
Add documentation.
ajnavarro Jun 30, 2022
c6fb6e5
docs: Routing.Routers
lidel Jul 3, 2022
0b5c9d4
Requested changes.
ajnavarro Jul 4, 2022
1ed2be0
Add some documentation on new fx functions.
ajnavarro Jul 4, 2022
32c6bba
Add changelog entry and integration tests
ajnavarro Jul 5, 2022
4bff8ff
test: sharness for 'dht' in 'routing' commands
lidel Jul 5, 2022
dc128d8
test(sharness): delegated routing via reframe URL
lidel Jul 6, 2022
bc00d2d
Add more tests for delegated routing.
ajnavarro Jul 6, 2022
43bc61e
If any put operation fails, the tiered router will fail.
ajnavarro Jul 6, 2022
6cb3a11
Merge master into feature/delegated-routing
lidel Jul 6, 2022
2f57adf
refactor: Routing.Routers: Parameters.Endpoint
lidel Jul 6, 2022
e3e2c6b
Try to improve CHANGELOG entry.
ajnavarro Jul 7, 2022
72e9546
chore: update reframe spec link
lidel Jul 7, 2022
4feb2ac
Update go-delegated-routing dependency
ajnavarro Jul 7, 2022
1e42022
Fix config error test
ajnavarro Jul 7, 2022
04014f5
use new changelog format
ajnavarro Jul 7, 2022
8020ba8
Remove port conflict
ajnavarro Jul 7, 2022
70d50fe
Merge branch 'master' into feature/delegated-routing
ajnavarro Jul 7, 2022
45a716e
go mod tidy
ajnavarro Jul 7, 2022
79ed919
ProviderManyWrapper to ProviderMany
ajnavarro Jul 7, 2022
1cf68ff
Update docs/changelogs/v0.14.md
ajnavarro Jul 7, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
# go-ipfs changelog

## v0.14.0 TBD

### Overview

- Delegated routing using [Reframe protocol](https://github.com/ipfs/specs/blob/main/REFRAME.md)
ajnavarro marked this conversation as resolved.
Show resolved Hide resolved

## v0.13.0 2022-05-04

We're happy to announce go-ipfs 0.13.0, packed full of changes and improvements!
Expand Down
5 changes: 1 addition & 4 deletions cmd/ipfs/daemon.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -400,10 +400,7 @@ func daemonFunc(req *cmds.Request, re cmds.ResponseEmitter, env cmds.Environment

routingOption, _ := req.Options[routingOptionKwd].(string)
if routingOption == routingOptionDefaultKwd {
routingOption = cfg.Routing.Type
if routingOption == "" {
routingOption = routingOptionDHTKwd
}
routingOption = cfg.Routing.Type.WithDefault(routingOptionDHTKwd)
}
switch routingOption {
case routingOptionSupernodeKwd:
Expand Down
2 changes: 1 addition & 1 deletion config/init.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ func InitWithIdentity(identity Identity) (*Config, error) {
},

Routing: Routing{
Type: "dht",
Type: NewOptionalString("dht"),
},

// setup the node mount points.
Expand Down
2 changes: 1 addition & 1 deletion config/profile.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,7 @@ functionality - performance of content discovery and data
fetching may be degraded.
`,
Transform: func(c *Config) error {
c.Routing.Type = "dhtclient"
c.Routing.Type = NewOptionalString("dhtclient")
c.AutoNAT.ServiceMode = AutoNATServiceDisabled
c.Reprovider.Interval = "0"

Expand Down
35 changes: 35 additions & 0 deletions config/routing.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,5 +5,40 @@ type Routing struct {
// Type sets default daemon routing mode.
//
// Can be one of "dht", "dhtclient", "dhtserver", "none", or unset.
Type *OptionalString `json:",omitempty"`

Routers map[string]Router
}

type Router struct {

// Currenly only supported Type is "reframe".
// Reframe type allows to add other resolvers using the Reframe spec:
// https://github.com/ipfs/specs/blob/master/REFRAME.md
lidel marked this conversation as resolved.
Show resolved Hide resolved
// In the future we will support "dht" and other Types here.
Type string

Enabled Flag `json:",omitempty"`

// Parameters are extra configuration that this router might need.
// A common one for reframe endpoints is "address".
Parameters map[string]string
}

// Type is the routing type.
// Depending of the type we need to instantiate different Routing implementations.
type RouterType string

const (
RouterTypeReframe RouterType = "reframe"
)

type RouterParam string

const (
// RouterParamAddress is the URL where the routing implementation will point to get the information.
// Usually used for reframe Routers.
RouterParamAddress RouterParam = "address"

RouterParamPriority RouterParam = "priority"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lidel idk if you have thoughts here, but I'm not sure how much we want to expose here.

Might be useful info to have to understand some notion of user preference, but also we may want to just do more in parallel depending on our concerns about resource usage.

AFAICT the only areas where priority kicks in are for the GetValue, GetPublicKey and FindPeer functions (https://github.com/libp2p/go-libp2p-routing-helpers/blob/506670d53a31503f257980dbf354b3ea1b990fb7/tiered.go#L30). GetPublicKey is basically unused, FindPeer could probably be in parallel without issue, and GetValue sort of makes sense to have prioritization (there's some weird edge-cases around IPNS records since they're mutable and you're trying to find the latest ones but don't want to hang around forever checking every system) but IIRC SearchValue is generally used anyway which has parallelism.

Copy link
Member

@lidel lidel Jul 3, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aschmahmann If we talk about future where we have DHT routers here as well, I think we need two optional settings:

  1. priority – allows setting the order (when it matters)
  2. mandatory – allows for marking some routers as mandatory and others as best-effort

With these, we will be able to model custom rules around parallelism and best-effort/mandatory routers in the future, but for now we can skip them in config (thus optional with implicit default).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we add the mandatory flag we must stop using routinghelpers.TieredRouter and use some new implementation that supports that.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ack, this needs a design first.
We can "mandatory" decision/discussion to kubo 0.15 or later.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add an issue to track it: #9083

)
5 changes: 5 additions & 0 deletions config/types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -321,6 +321,11 @@ type OptionalString struct {
value *string
}

// NewOptionalString returns an OptionalString from a string
func NewOptionalString(s string) *OptionalString {
return &OptionalString{value: &s}
}

// WithDefault resolves the integer with the given default.
func (p *OptionalString) WithDefault(defaultValue string) (value string) {
if p == nil || p.value == nil {
Expand Down
6 changes: 6 additions & 0 deletions core/commands/commands_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,12 @@ func TestCommands(t *testing.T) {
"/dht/provide",
"/dht/put",
"/dht/query",
"/routing",
"/routing/put",
"/routing/get",
"/routing/findpeer",
"/routing/findprovs",
"/routing/provide",
"/diag",
"/diag/cmds",
"/diag/cmds/clear",
Expand Down
Loading