Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

go-libp2p-kad-dht version 2 #864

Closed
wants to merge 65 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
7d18e89
Start go-libp2p-kad-dht v2 package
dennis-tra Aug 14, 2023
5bec109
improve DHT struct field documentation
dennis-tra Aug 14, 2023
311f0d3
add: default bootstrap peers method
dennis-tra Aug 14, 2023
dfe3a81
go mod tidy
dennis-tra Aug 14, 2023
57c4988
use slog for logging
dennis-tra Aug 14, 2023
984f28e
handle reachability change events
dennis-tra Aug 14, 2023
35a977f
Add network event subscription
dennis-tra Aug 15, 2023
e8e5be2
Add stream handler implementation
dennis-tra Aug 15, 2023
027c8d2
Add FIND_NODE and PING handlers
dennis-tra Aug 16, 2023
3171894
add handlers and tests
dennis-tra Aug 18, 2023
9f8f4a7
mark triage and gokad parts
dennis-tra Aug 18, 2023
3cdfb34
test put value handler
dennis-tra Aug 18, 2023
c584b02
add get value tests
dennis-tra Aug 18, 2023
4dfcd8b
remove unused methods
dennis-tra Aug 18, 2023
0af762c
update handlers
dennis-tra Aug 21, 2023
741843b
introduce backend concept
dennis-tra Aug 23, 2023
5e09e95
improve backend documentation
dennis-tra Aug 23, 2023
619429c
introduce protocol constants
dennis-tra Aug 23, 2023
3648184
simplify key handling
dennis-tra Aug 23, 2023
015617e
improve documentation
dennis-tra Aug 23, 2023
38c2d32
add add providers tests
dennis-tra Aug 23, 2023
7a0e26c
improve telemetry
dennis-tra Aug 23, 2023
a8a222b
Add AddressFilter feature
dennis-tra Aug 24, 2023
6bdd7ec
WIP
dennis-tra Aug 24, 2023
14c478c
test backend provider garbage collection
dennis-tra Aug 24, 2023
2643b72
refactor provider backend tests
dennis-tra Aug 24, 2023
890d4ec
add stream tests
dennis-tra Aug 24, 2023
bdac291
add logErr helper method
dennis-tra Aug 24, 2023
50f8d00
babysteps towards routing.Routing and kademlia.Router
dennis-tra Aug 24, 2023
5a30d89
update deps
dennis-tra Aug 24, 2023
4660bbc
add test for unregistered message type
dennis-tra Aug 25, 2023
9f88f20
add comments
dennis-tra Aug 25, 2023
9559064
clean up backends in DHT close
dennis-tra Aug 25, 2023
4418cfb
add notifee test
dennis-tra Aug 25, 2023
1cbb419
moved two handler tests from v1
dennis-tra Aug 28, 2023
57c5df3
Move kademlia types to own package
dennis-tra Aug 30, 2023
99dbdc6
Add backend tracing
dennis-tra Sep 1, 2023
f16d79e
improve kadt package doc
dennis-tra Sep 1, 2023
2610f89
document tracedBackend
dennis-tra Sep 1, 2023
54f20b5
Integrate Zikade/go-kademlia in v2 (#880)
dennis-tra Sep 7, 2023
722e958
Simplify usage of kadtest.CtxShort
iand Sep 8, 2023
add1d60
Merge pull request #885 from libp2p/v2-ctx-short
iand Sep 8, 2023
7d838c5
Give dht and coordinator their own telemetry instances (#891)
iand Sep 11, 2023
77dbff0
Migrate go-kademlia state machines (#893)
iand Sep 13, 2023
97e4e02
v2: upgrade build to include go1.21 (#890)
iand Sep 16, 2023
1d35505
Test query interactions with routing table (#887)
iand Sep 18, 2023
3723b8a
remove jaeger dependency (#900)
dennis-tra Sep 18, 2023
156aab2
fix: avoid panic when node is re-added to probe list (#902)
dennis-tra Sep 18, 2023
1220ddd
Decouple coord package from addressing (#903)
iand Sep 19, 2023
e86381e
refactor: v2 simplify tracing (#924)
dennis-tra Sep 19, 2023
83329a4
Clean up DHT test helpers (#928)
iand Sep 19, 2023
2da54ab
Improve query capabilities (#932)
iand Sep 21, 2023
74ffa67
Add broadcast state machine for storing records in the DHT (#930)
dennis-tra Sep 22, 2023
ae5a094
Add explore state machine to expand population of routing table (#934)
iand Sep 25, 2023
09dd7b0
Expose behaviour and state machine configs (#937)
iand Sep 25, 2023
dd5e537
feat: findProvidersAsync (#938)
dennis-tra Sep 26, 2023
dedca86
Add metrics to routing state machines (#939)
iand Sep 27, 2023
e4b1034
add provide routing tests (#940)
dennis-tra Sep 27, 2023
4fa560f
fix: bootstrap state machine goes idle after completion (#943)
iand Sep 27, 2023
6a4249c
Logging improvements (#941)
iand Sep 28, 2023
03adce6
Implement SearchValue/GetValue (#942)
dennis-tra Sep 28, 2023
90d748b
fix: missing QueryMessage return value
dennis-tra Sep 28, 2023
0e628c0
fix: use correct err log method
dennis-tra Sep 28, 2023
509eee4
fix: avoid panic in bootstrap when late messages arrive (#949)
iand Sep 28, 2023
1d1fe93
Use go-libdht (#952)
iand Sep 29, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/uci.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
files: # Configure what Unified CI templates should be used for your repository; defaults to primary language default fileset
- .github/workflows/go-check.yml
- .github/workflows/go-test.yml
force: true # Configure whether Unified CI should overwrite existing workflows; defaults to false
56 changes: 48 additions & 8 deletions .github/workflows/go-check.yml
Original file line number Diff line number Diff line change
@@ -1,9 +1,23 @@
# File managed by web3-bot. DO NOT EDIT.
# See https://github.com/protocol/.github/ for details.

on: [push, pull_request]
name: Go Checks

defaults:
run:
working-directory: ./v2


on:
pull_request:
push:
branches: ["main"]
workflow_dispatch:

permissions:
contents: read

concurrency:
group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event_name == 'push' && github.sha || github.ref }}
cancel-in-progress: true

jobs:
unit:
runs-on: ubuntu-latest
Expand All @@ -13,15 +27,37 @@ jobs:
with:
submodules: recursive
- id: config
uses: protocol/.github/.github/actions/read-config@master
- uses: actions/setup-go@v3
uses: pl-strflt/uci/.github/actions/read-config@main
- id: go-mod
uses: pl-strflt/uci/.github/actions/read-go-mod@main
- id: go
uses: actions/setup-go@v4
with:
go-version: 1.20.x
go-version: '1.20.8'
cache: false
- name: Run repo-specific setup
uses: ./.github/actions/go-check-setup
if: hashFiles('./.github/actions/go-check-setup') != ''
- name: Install staticcheck
run: go install honnef.co/go/tools/cmd/staticcheck@4970552d932f48b71485287748246cf3237cebdf # 2023.1 (v0.4.0)
env:
STATICCHECK_VERSIONS: |
{
"1.21": "9e12e6014d3b0a854950490051ad1338fc6badd1",
"1.20": "9e12e6014d3b0a854950490051ad1338fc6badd1",
"1.19": "376210a89477dedbe6fdc4484b233998650d7b3c",
"1.18": "376210a89477dedbe6fdc4484b233998650d7b3c",
"1.17": "c8caa92bad8c27ae734c6725b8a04932d54a147b",
"1.16": "4dc1992c9bb4310ba1e98b30c8d7d46444891d3b",
"1.15": "5b7de96f09104e2be384aa93a7c821eb5e77378b",
"1.14": "5b7de96f09104e2be384aa93a7c821eb5e77378b",
"1.13": "afd67930eec2a9ed3e9b19f684d17a062285f16a"
}
GO_VERSION: ${{ steps.go.outputs.go-version }}
GO111MODULE: on
run: |
version="$(jq -nr 'env.STATICCHECK_VERSIONS | fromjson | .[env.GO_VERSION | sub("\\.[^.]+$"; "")] // "latest"')"
echo "Installing staticcheck@$version"
go install honnef.co/go/tools/cmd/staticcheck@$version || go get honnef.co/go/tools/cmd/staticcheck@$version
- name: Check that go.mod is tidy
uses: protocol/multiple-go-modules@v1.2
with:
Expand All @@ -32,6 +68,7 @@ jobs:
exit 1
fi
git diff --exit-code -- go.sum go.mod
working-directory: ./v2
- name: gofmt
if: success() || failure() # run this step even if the previous one failed
run: |
Expand All @@ -45,13 +82,15 @@ jobs:
uses: protocol/multiple-go-modules@v1.2
with:
run: go vet ./...
working-directory: ./v2
- name: staticcheck
if: success() || failure() # run this step even if the previous one failed
uses: protocol/multiple-go-modules@v1.2
with:
run: |
set -o pipefail
staticcheck ./... | sed -e 's@\(.*\)\.go@./\1.go@g'
working-directory: ./v2
- name: go generate
uses: protocol/multiple-go-modules@v1.2
if: (success() || failure()) && fromJSON(steps.config.outputs.json).gogenerate == true
Expand All @@ -65,3 +104,4 @@ jobs:
git status --short
exit 1
fi
working-directory: ./v2
95 changes: 68 additions & 27 deletions .github/workflows/go-test.yml
Original file line number Diff line number Diff line change
@@ -1,52 +1,87 @@
# File managed by web3-bot. DO NOT EDIT.
# See https://github.com/protocol/.github/ for details.

on: [push, pull_request]
name: Go Test

on:
pull_request:
push:
branches: ["main"]
workflow_dispatch:

permissions:
contents: read

concurrency:
group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event_name == 'push' && github.sha || github.ref }}
cancel-in-progress: true

jobs:
unit:
strategy:
fail-fast: false
matrix:
os: [ "ubuntu", "windows", "macos" ]
go: ["1.19.x","1.20.x"]
go: ["1.20.8", "1.21.1"]
env:
COVERAGES: ""
GOTESTFLAGS: -cover -coverprofile=module-coverage.txt -coverpkg=./...
GO386FLAGS: ''
GORACEFLAGS: ''
runs-on: ${{ fromJSON(vars[format('UCI_GO_TEST_RUNNER_{0}', matrix.os)] || format('"{0}-latest"', matrix.os)) }}
name: ${{ matrix.os }} (go ${{ matrix.go }})
steps:
- name: Use msys2 on windows
if: matrix.os == 'windows'
# The executable for msys2 is also called bash.cmd
# https://github.com/actions/virtual-environments/blob/main/images/win/Windows2019-Readme.md#shells
# If we prepend its location to the PATH
# subsequent 'shell: bash' steps will use msys2 instead of gitbash
run: echo "C:/msys64/usr/bin" >> $GITHUB_PATH
- uses: actions/checkout@v3
with:
submodules: recursive
# Update to v4 is blocked by https://github.com/actions/setup-go/pull/411
# - uses: actions/setup-go@v4
- uses: actions/setup-go@v3
with:
go-version: ${{ steps.go.outputs.version }}
# cache: false
- id: config
uses: protocol/.github/.github/actions/read-config@master
uses: pl-strflt/uci/.github/actions/read-config@main
- id: go-mod
uses: pl-strflt/uci/.github/actions/read-go-mod@main
- id: go
uses: actions/setup-go@v4
with:
go-version: ${{ steps.go.outputs.version }}
cache: false
- if: toJSON(fromJSON(steps.config.outputs.json).shuffle) != 'false'
run: |
echo "GOTESTFLAGS=-shuffle=on $GOTESTFLAGS" >> $GITHUB_ENV
echo "GO386FLAGS=-shuffle=on $GO386FLAGS" >> $GITHUB_ENV
echo "GORACEFLAGS=-shuffle=on $GORACEFLAGS" >> $GITHUB_ENV
- if: toJSON(fromJSON(steps.config.outputs.json).verbose) != 'false'
run: |
echo "GOTESTFLAGS=-v $GOTESTFLAGS" >> $GITHUB_ENV
echo "GO386FLAGS=-v $GO386FLAGS" >> $GITHUB_ENV
echo "GORACEFLAGS=-v $GORACEFLAGS" >> $GITHUB_ENV
# Update to v4 is blocked by https://github.com/actions/setup-go/pull/411
# - uses: actions/setup-go@v4
- uses: actions/setup-go@v3
with:
go-version: ${{ matrix.go }}
go-version: ${{ steps.go.outputs.version }}
- name: Go information
run: |
go version
go env
- name: Use msys2 on windows
if: matrix.os == 'windows'
shell: bash
# The executable for msys2 is also called bash.cmd
# https://github.com/actions/virtual-environments/blob/main/images/win/Windows2019-Readme.md#shells
# If we prepend its location to the PATH
# subsequent 'shell: bash' steps will use msys2 instead of gitbash
run: echo "C:/msys64/usr/bin" >> $GITHUB_PATH
- name: Run repo-specific setup
uses: ./.github/actions/go-test-setup
if: hashFiles('./.github/actions/go-test-setup') != ''
- name: Run tests
if: contains(fromJSON(steps.config.outputs.json).skipOSes, matrix.os) == false
uses: protocol/multiple-go-modules@v1.2
env:
GOFLAGS: ${{ format('{0} {1}', env.GOTESTFLAGS, env.GOFLAGS) }}
with:
# Use -coverpkg=./..., so that we include cross-package coverage.
# If package ./A imports ./B, and ./A's tests also cover ./B,
# this means ./B's coverage will be significantly higher than 0%.
run: go test -v -shuffle=on -coverprofile=module-coverage.txt -coverpkg=./... ./...
run: go test ./...
working-directory: ./v2
- name: Run tests (32 bit)
# can't run 32 bit tests on OSX.
if: matrix.os != 'macos' &&
Expand All @@ -55,22 +90,28 @@ jobs:
uses: protocol/multiple-go-modules@v1.2
env:
GOARCH: 386
GOFLAGS: ${{ format('{0} {1}', env.GO386FLAGS, env.GOFLAGS) }}
with:
run: |
export "PATH=$PATH_386:$PATH"
go test -v -shuffle=on ./...
go test ./...
working-directory: ./v2
- name: Run tests with race detector
# speed things up. Windows and OSX VMs are slow
if: matrix.os == 'ubuntu' &&
fromJSON(steps.config.outputs.json).skipRace != true &&
contains(fromJSON(steps.config.outputs.json).skipOSes, matrix.os) == false
uses: protocol/multiple-go-modules@v1.2
env:
GOFLAGS: ${{ format('{0} {1}', env.GORACEFLAGS, env.GOFLAGS) }}
with:
run: go test -v -race ./...
run: go test -race ./...
working-directory: ./v2
- name: Collect coverage files
shell: bash
run: echo "COVERAGES=$(find . -type f -name 'module-coverage.txt' | tr -s '\n' ',' | sed 's/,$//')" >> $GITHUB_ENV
id: coverages
run: echo "files=$(find . -type f -name 'module-coverage.txt' | tr -s '\n' ',' | sed 's/,$//')" >> $GITHUB_OUTPUT
- name: Upload coverage to Codecov
uses: codecov/codecov-action@d9f34f8cd5cb3b3eb79b3e4b5dae3a16df499a70 # v3.1.1
uses: codecov/codecov-action@eaaf4bedf32dbdc6b720b63067d99c4d77d6047d # v3.1.4
with:
files: '${{ env.COVERAGES }}'
env_vars: OS=${{ matrix.os }}, GO=${{ matrix.go }}
files: ${{ steps.coverages.outputs.files }}
env_vars: OS=${{ matrix.os }}, GO=${{ steps.go.outputs.version }}
18 changes: 12 additions & 6 deletions .github/workflows/release-check.yml
Original file line number Diff line number Diff line change
@@ -1,13 +1,19 @@
# File managed by web3-bot. DO NOT EDIT.
# See https://github.com/protocol/.github/ for details.

name: Release Checker

on:
pull_request_target:
paths: [ 'version.json' ]
types: [ opened, synchronize, reopened, labeled, unlabeled ]
workflow_dispatch:

permissions:
contents: write
pull-requests: write

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
release-check:
uses: protocol/.github/.github/workflows/release-check.yml@master
with:
go-version: 1.20.x
uses: pl-strflt/uci/.github/workflows/release-check.yml@v0.0
14 changes: 10 additions & 4 deletions .github/workflows/releaser.yml
Original file line number Diff line number Diff line change
@@ -1,11 +1,17 @@
# File managed by web3-bot. DO NOT EDIT.
# See https://github.com/protocol/.github/ for details.

name: Releaser

on:
push:
paths: [ 'version.json' ]
workflow_dispatch:

permissions:
contents: write

concurrency:
group: ${{ github.workflow }}-${{ github.sha }}
cancel-in-progress: true

jobs:
releaser:
uses: protocol/.github/.github/workflows/releaser.yml@master
uses: pl-strflt/uci/.github/workflows/releaser.yml@v0.0
14 changes: 10 additions & 4 deletions .github/workflows/tagpush.yml
Original file line number Diff line number Diff line change
@@ -1,12 +1,18 @@
# File managed by web3-bot. DO NOT EDIT.
# See https://github.com/protocol/.github/ for details.

name: Tag Push Checker

on:
push:
tags:
- v*

permissions:
contents: read
issues: write

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
releaser:
uses: protocol/.github/.github/workflows/tagpush.yml@master
uses: pl-strflt/uci/.github/workflows/tagpush.yml@v0.0
3 changes: 3 additions & 0 deletions CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
Expand Up @@ -13,3 +13,6 @@
# records are IPFS specific
/records.go @libp2p/kubo-maintainers @guillaumemichel
/records_test.go @libp2p/kubo-maintainers @guillaumemichel


/v2/ @dennis-tra @iand
7 changes: 6 additions & 1 deletion dht_bootstrap.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,14 @@ var DefaultBootstrapPeers []multiaddr.Multiaddr

// Minimum number of peers in the routing table. If we drop below this and we
// see a new peer, we trigger a bootstrap round.
// GOKAD: config belongs to go-kademlia
var minRTRefreshThreshold = 10

const (
// GOKAD: config belongs to go-kademlia
periodicBootstrapInterval = 2 * time.Minute
maxNBoostrappers = 2
// GOKAD: config belongs to go-kademlia
maxNBoostrappers = 2
)

func init() {
Expand Down Expand Up @@ -67,6 +70,7 @@ func (dht *IpfsDHT) Bootstrap(ctx context.Context) error {
//
// The returned channel will block until the refresh finishes, then yield the
// error and close. The channel is buffered and safe to ignore.
// TRIAGE: needed or alternative?
func (dht *IpfsDHT) RefreshRoutingTable() <-chan error {
return dht.rtRefreshManager.Refresh(false)
}
Expand All @@ -76,6 +80,7 @@ func (dht *IpfsDHT) RefreshRoutingTable() <-chan error {
//
// The returned channel will block until the refresh finishes, then yield the
// error and close. The channel is buffered and safe to ignore.
// TRIAGE: needed or alternative?
func (dht *IpfsDHT) ForceRefresh() <-chan error {
return dht.rtRefreshManager.Refresh(true)
}
1 change: 1 addition & 0 deletions v2/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
github.com
Loading