Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add script to collect DHT metrics #3625

Merged
merged 45 commits into from
Sep 7, 2022
Merged

Add script to collect DHT metrics #3625

merged 45 commits into from
Sep 7, 2022
+591 −14

Conversation

shyba
Copy link
Member

@shyba shyba commented Jun 27, 2022

No description provided.

@shyba shyba added type: task Either work that's not related to the code, or a small chore that does not fit into other categories area: DHT labels Jun 27, 2022
@shyba shyba force-pushed the dht_crawler branch 2 times, most recently from 5c07fcb to d1e6c90 Compare July 8, 2022 06:06
@shyba shyba mentioned this pull request Jul 11, 2022
Closed
@shyba shyba force-pushed the dht_crawler branch 3 times, most recently from cc6cdc0 to bfbaf14 Compare July 27, 2022 11:37
@shyba shyba requested a review from jackrobison August 3, 2022 13:53
@lbry-bot lbry-bot assigned jackrobison and unassigned shyba Aug 3, 2022
@coveralls
Copy link

coveralls commented Aug 12, 2022 •
edited
Loading

Coverage Status

Coverage remained the same at 57.899% when pulling c7c2d6f on dht_crawler into a334a93 on master.

shyba added 15 commits September 7, 2022 12:03
shyba added 27 commits September 7, 2022 12:03
@shyba
dht_crawler: fix node id store
cd42f0d
@shyba
dht_crawler: keep working set in memory, flush to db on intervals
508bdb8
@shyba
dht_crawler: skip ping if known node_id
0497698
@shyba
dht_crawler: serve prometheus metrics at 7070
cfe5c8d
@shyba
dht_crawler: only count non zero connections
62eb9d5
@shyba
dht_crawler: randomize port when idle
85ff487
@shyba
update pip and setuptools on dht dockerfile
4a3a7e3
@shyba
change default metric port, add sqlalchemy to dockerfile
47a5d37
@shyba
refactor script, remove dep
13af780
@shyba
add sd_hash prober
0d6125d
@shyba
add script to generate probe dataset
e2922a4
@shyba
format logging
43b45a9
@shyba
add arg for db path
9aa9ecd
@shyba
dht crawler:improve logging, metrics, make startup concurrent
df77392
@shyba
skip peers with bad ports without raising
da2ffb0
@shyba
docker: add volume declaration
b5c390c
@shyba
dht_crawler: fix logging for missing ports
cc64789
@shyba
fix hosting metrics, improve logging
5e58c2f
@shyba
dht_crawler:only count latency during findNode
f077e56
@shyba
dht_crawler: gather both loops, avoid task exceptions being hidden
c38573d
@shyba
dht_crawler: skip saving connections for now
d0497cf
@shyba
dht_crawler: clean in memory set for expired peers
0e7a1ae
@shyba
dht_crawler: wait and retry during port switch
ab67f41
@shyba
use a histogram for latency, remove labels
0b059a5
@shyba
routing table sizes as histogram
8d9d2c7
@shyba
fix crawler startup query
c6c0228
@shyba
collect connections reachability
c7c2d6f
@shyba shyba merged commit f745560 into master Sep 7, 2022
@shyba shyba deleted the dht_crawler branch September 7, 2022 15:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jackrobison jackrobison Awaiting requested review from jackrobison

Assignees

@jackrobison jackrobison

Labels
area: DHT type: task Either work that's not related to the code, or a small chore that does not fit into other categories
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@shyba @coveralls @jackrobison