Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

added command to verify remote manifests from webseeds #9762

Merged
merged 13 commits into from
Mar 24, 2024

Conversation

awskii
Copy link
Member

@awskii awskii commented Mar 19, 2024

Cherry picked initial commit against devel. Beginning of discussion is here #9750

@awskii
Copy link
Member Author

awskii commented Mar 19, 2024

Current output looks like

## REPORT on https://erigon2-v3-snapshots-sepolia.erigon.network
 - manifest exist: true
 - missing torrents (files without torrents): 0
 - dangling (data file not found) torrents: 5
 - invalid ETags format: 21/192
 - ETag fetch failed: 0/192
# Dangling torrents
domain/v1-storage.0-64.kv.torrent
idx/v1-accounts.0-64.ef.torrent
idx/v1-code.0-64.ef.torrent
idx/v1-logtopics.0-64.ef.torrent
idx/v1-tracesfrom.0-64.ef.torrent

# Invalid ETags format
domain/v1-accounts.0-64.kv                         4e13b4d39dd2a814ad553ba01a8d3d51-192
domain/v1-code.0-64.kv                             c34b7da1f76a137042ffc607eca98d5b-2229
domain/v1-commitment.0-64.kv                       41f1d579c57fa998a8398c0d2e60b3cb-2169
history/v1-accounts.0-64.v                         3ae170fd67bc6f6ddcf9ee911d70867f-1250
history/v1-storage.0-64.v                          a8ffc3d35d34bc07ba727b0444bafc6f-491
idx/v1-storage.0-64.ef                             f3be975948afc3790a57b88a2e93e045-3219
v1-001500-002000-transactions.seg                  374d84bab0deba57cfc6539785b24741-85
v1-002000-002500-transactions.seg                  0a4a059adc5fb3bd42cc4613741512b0-101
v1-002500-003000-transactions.seg                  7758c576ec3da6cbe01a8705c3b3caca-101
v1-003000-003500-transactions.seg                  43a48f09307287b5d38492d0465455ff-1415
v1-003500-004000-transactions.seg                  92e1aa02e53d9924c56f000ab3ed79be-5842
v1-004000-004100-transactions.seg                  3b33fee204ba8617d46cc5c45ad7cd9e-2189
v1-004100-004200-transactions.seg                  72596cdcdca0cced80282abeeea5346b-1946
v1-004200-004300-transactions.seg                  1e90c1d899425923ee82950ae97ab5d3-1522
v1-004300-004400-transactions.seg                  6cde71e8c8b2fa80e32350a5efe92c4e-2025
v1-004400-004500-transactions.seg                  713e339d74a91946d8b1bbe23976589d-1567
v1-004500-004600-transactions.seg                  e3e0b63d6a9bf3db4880ce925c61ec6a-1113
v1-004600-004700-transactions.seg                  8e3a2c4e6c62380c327826e34d06bdfe-1135
v1-004700-004800-transactions.seg                  c5b58b7f416f32ae295968cc6d4ea438-1245
v1-004800-004900-transactions.seg                  56177e381fb19ddeee961089e0620f69-1634
v1-004900-005000-transactions.seg                  07fada8ca9427297922bc90cf95b197a-1706

== BAD https://erigon2-v3-snapshots-sepolia.erigon.networkn
## Seed https://erigon3-v1-snapshots-sepolia.erigon.network manifest.txt fetch failed: 404 Not Found
## REPORT on https://erigon3-v1-snapshots-sepolia.erigon.network: manifest not found

WARN[03-21|14:12:36.276] [snapshots.webseed] verify manifest      err="webseed.http: status=404, url=https://erigon3-v1-snapshots-sepolia.erigon.network/manifest.txt"
## REPORT on https://erigon3-v3-snapshots-sepolia.erigon.network
 - manifest exist: true
 - missing torrents (files without torrents): 224
 - dangling (data file not found) torrents: 0
 - invalid ETags format: 48/394
 - ETag fetch failed: 50/394

# Missing torrents
accessors/v1-accounts.0-64.efi
accessors/v1-accounts.0-64.vi
accessors/v1-accounts.64-72.efi
accessors/v1-accounts.64-72.vi
accessors/v1-accounts.64-80.efi
accessors/v1-accounts.64-80.vi
accessors/v1-accounts.72-76.efi
accessors/v1-accounts.72-76.vi
accessors/v1-accounts.76-78.efi
accessors/v1-accounts.76-78.vi
accessors/v1-accounts.78-79.efi
accessors/v1-accounts.78-79.vi
accessors/v1-accounts.79-80.efi
accessors/v1-accounts.79-80.vi
accessors/v1-accounts.80-82.efi
accessors/v1-accounts.80-82.vi
accessors/v1-code.0-64.efi
accessors/v1-code.0-64.vi
accessors/v1-code.64-80.efi
accessors/v1-code.64-80.vi
accessors/v1-code.80-82.efi
accessors/v1-code.80-82.vi
accessors/v1-logaddrs.0-64.efi
accessors/v1-logaddrs.64-80.efi
accessors/v1-logaddrs.80-82.efi
accessors/v1-logtopics.0-64.efi
accessors/v1-logtopics.64-80.efi
accessors/v1-logtopics.80-82.efi
accessors/v1-storage.0-64.efi
accessors/v1-storage.0-64.vi
accessors/v1-storage.64-72.efi
accessors/v1-storage.64-72.vi
accessors/v1-storage.64-80.efi
accessors/v1-storage.64-80.vi
accessors/v1-storage.72-76.efi
accessors/v1-storage.72-76.vi
accessors/v1-storage.76-78.efi
accessors/v1-storage.76-78.vi
accessors/v1-storage.78-79.efi
accessors/v1-storage.78-79.vi
accessors/v1-storage.79-80.efi
accessors/v1-storage.79-80.vi
accessors/v1-storage.80-82.efi
accessors/v1-storage.80-82.vi
accessors/v1-tracesfrom.0-64.efi
accessors/v1-tracesfrom.64-80.efi
accessors/v1-tracesfrom.80-82.efi
accessors/v1-tracesto.0-64.efi
accessors/v1-tracesto.64-80.efi
accessors/v1-tracesto.80-82.efi
domain/v1-accounts.0-64.bt
domain/v1-accounts.0-64.kvei
domain/v1-accounts.64-72.bt
domain/v1-accounts.64-72.kv
domain/v1-accounts.64-72.kvei
domain/v1-accounts.64-80.bt
domain/v1-accounts.64-80.kv
domain/v1-accounts.64-80.kvei
domain/v1-accounts.72-76.bt
domain/v1-accounts.72-76.kv
domain/v1-accounts.72-76.kvei
domain/v1-accounts.76-78.bt
domain/v1-accounts.76-78.kv
domain/v1-accounts.76-78.kvei
domain/v1-accounts.78-79.bt
domain/v1-accounts.78-79.kv
domain/v1-accounts.78-79.kvei
domain/v1-accounts.79-80.bt
domain/v1-accounts.79-80.kv
domain/v1-accounts.79-80.kvei
domain/v1-accounts.80-82.bt
domain/v1-accounts.80-82.kv
domain/v1-accounts.80-82.kvei
domain/v1-code.0-64.bt
domain/v1-code.0-64.kvei
domain/v1-code.64-80.bt
domain/v1-code.64-80.kv
domain/v1-code.64-80.kvei
domain/v1-code.80-82.bt
domain/v1-code.80-82.kv
domain/v1-code.80-82.kvei
domain/v1-commitment.0-64.bt
domain/v1-commitment.0-64.kvei
domain/v1-commitment.64-80.bt
domain/v1-commitment.64-80.kv
domain/v1-commitment.64-80.kvei
domain/v1-commitment.80-82.bt
domain/v1-commitment.80-82.kv
domain/v1-commitment.80-82.kvei
domain/v1-storage.0-64.bt
domain/v1-storage.0-64.kvei
domain/v1-storage.64-72.bt
domain/v1-storage.64-72.kv
domain/v1-storage.64-72.kvei
domain/v1-storage.64-80.bt
domain/v1-storage.64-80.kv
domain/v1-storage.64-80.kvei
domain/v1-storage.72-76.bt
domain/v1-storage.72-76.kv
domain/v1-storage.72-76.kvei
domain/v1-storage.76-78.bt
domain/v1-storage.76-78.kv
domain/v1-storage.76-78.kvei
domain/v1-storage.78-79.bt
domain/v1-storage.78-79.kv
domain/v1-storage.78-79.kvei
domain/v1-storage.79-80.bt
domain/v1-storage.79-80.kv
domain/v1-storage.79-80.kvei
domain/v1-storage.80-82.bt
domain/v1-storage.80-82.kv
domain/v1-storage.80-82.kvei
history/v1-accounts.64-72.v
history/v1-accounts.64-80.v
history/v1-accounts.72-76.v
history/v1-accounts.76-78.v
history/v1-accounts.78-79.v
history/v1-accounts.79-80.v
history/v1-accounts.80-82.v
history/v1-code.64-80.v
history/v1-code.80-82.v
history/v1-storage.64-72.v
history/v1-storage.64-80.v
history/v1-storage.72-76.v
history/v1-storage.76-78.v
history/v1-storage.78-79.v
history/v1-storage.79-80.v
history/v1-storage.80-82.v
idx/v1-accounts.64-72.ef
idx/v1-accounts.64-80.ef
idx/v1-accounts.72-76.ef
idx/v1-accounts.76-78.ef
idx/v1-accounts.78-79.ef
idx/v1-accounts.79-80.ef
idx/v1-accounts.80-82.ef
idx/v1-code.64-80.ef
idx/v1-code.80-82.ef
idx/v1-logaddrs.64-80.ef
idx/v1-logaddrs.80-82.ef
idx/v1-logtopics.64-80.ef
idx/v1-logtopics.80-82.ef
idx/v1-storage.64-72.ef
idx/v1-storage.64-80.ef
idx/v1-storage.72-76.ef
idx/v1-storage.76-78.ef
idx/v1-storage.78-79.ef
idx/v1-storage.79-80.ef
idx/v1-storage.80-82.ef
idx/v1-tracesfrom.64-80.ef
idx/v1-tracesfrom.80-82.ef
idx/v1-tracesto.64-80.ef
idx/v1-tracesto.80-82.ef
v1-000000-000500-bodies.idx
v1-000000-000500-headers.idx
v1-000000-000500-transactions-to-block.idx
v1-000000-000500-transactions.idx
v1-000500-001000-bodies.idx
v1-000500-001000-headers.idx
v1-000500-001000-transactions-to-block.idx
v1-000500-001000-transactions.idx
v1-001000-001500-bodies.idx
v1-001000-001500-headers.idx
v1-001000-001500-transactions-to-block.idx
v1-001000-001500-transactions.idx
v1-001500-002000-bodies.idx
v1-001500-002000-headers.idx
v1-001500-002000-transactions-to-block.idx
v1-001500-002000-transactions.idx
v1-002000-002500-bodies.idx
v1-002000-002500-headers.idx
v1-002000-002500-transactions-to-block.idx
v1-002000-002500-transactions.idx
v1-002500-003000-bodies.idx
v1-002500-003000-headers.idx
v1-002500-003000-transactions-to-block.idx
v1-002500-003000-transactions.idx
v1-003000-003500-bodies.idx
v1-003000-003500-headers.idx
v1-003000-003500-transactions-to-block.idx
v1-003000-003500-transactions.idx
v1-003500-004000-bodies.idx
v1-003500-004000-headers.idx
v1-003500-004000-transactions-to-block.idx
v1-003500-004000-transactions.idx
v1-004000-004100-bodies.idx
v1-004000-004100-headers.idx
v1-004000-004100-transactions-to-block.idx
v1-004000-004100-transactions.idx
v1-004100-004200-bodies.idx
v1-004100-004200-headers.idx
v1-004100-004200-transactions-to-block.idx
v1-004100-004200-transactions.idx
v1-004200-004300-bodies.idx
v1-004200-004300-headers.idx
v1-004200-004300-transactions-to-block.idx
v1-004200-004300-transactions.idx
v1-004300-004400-bodies.idx
v1-004300-004400-headers.idx
v1-004300-004400-transactions-to-block.idx
v1-004300-004400-transactions.idx
v1-004400-004500-bodies.idx
v1-004400-004500-headers.idx
v1-004400-004500-transactions-to-block.idx
v1-004400-004500-transactions.idx
v1-004500-004600-bodies.idx
v1-004500-004600-headers.idx
v1-004500-004600-transactions-to-block.idx
v1-004500-004600-transactions.idx
v1-004600-004700-bodies.idx
v1-004600-004700-headers.idx
v1-004600-004700-transactions-to-block.idx
v1-004600-004700-transactions.idx
v1-004700-004800-bodies.idx
v1-004700-004800-headers.idx
v1-004700-004800-transactions-to-block.idx
v1-004700-004800-transactions.idx
v1-004800-004900-bodies.idx
v1-004800-004900-headers.idx
v1-004800-004900-transactions-to-block.idx
v1-004800-004900-transactions.idx
v1-004900-005000-bodies.idx
v1-004900-005000-headers.idx
v1-004900-005000-transactions-to-block.idx
v1-004900-005000-transactions.idx

# Invalid ETags format
domain/v1-accounts.0-64.kv                         08aa9964c4bdcd22036e85fb45b4a3f5-192
domain/v1-code.0-64.kv                             64141e02ef90b48aebbecced27ccb1f2-3
domain/v1-code.64-80.kv                            14023261d76507f86ccafecae9ec1669-308
domain/v1-commitment.0-64.kv                       497ec7d707168559b6531112cc6097d4-3
domain/v1-commitment.64-80.kv                      ecebcb2afaaa4327da972ea20cc7d98d-967
domain/v1-commitment.80-82.kv                      03d86886fd3165ba1474316fb75aace6-275
domain/v1-storage.0-64.kv                          d127e5dc473a15b647872672d19fde73-2
domain/v1-storage.64-72.kv                         02f21db6d9680069cb776e54c5b45fe1-244
domain/v1-storage.64-80.kv                         f3b1d1e98ec8aa30b2fcaee32d2b2e7d-478
domain/v1-storage.72-76.kv                         a9d09ad7c0b7b879de923c18b8fd8605-140
domain/v1-storage.76-78.kv                         03080f8252f0fc356c12ef6fe0cbae30-79
domain/v1-storage.80-82.kv                         b5617e1599972bb245d05eb99268e35d-68
history/v1-accounts.0-64.v                         d8eb7111c38ea464b8d003e876989bec-2
history/v1-accounts.64-72.v                        8dfc49e6039fa5732e1278eff27b1601-171
history/v1-accounts.64-80.v                        3008f74456e8506af0556ca2919fc445-366
history/v1-accounts.72-76.v                        c9bfc468b7c7d26706b33253a8c13a35-91
history/v1-storage.0-64.v                          a8ffc3d35d34bc07ba727b0444bafc6f-491
history/v1-storage.64-72.v                         bbe2a9ca1158841e10b8bb04dcea5b00-68
history/v1-storage.64-80.v                         a64359498f562cc539bdc7125f6b5b9f-140
idx/v1-accounts.0-64.ef                            96225b5408bd82bde1d0ec6ae7dd0b3e-429
idx/v1-code.0-64.ef                                4bb295e5b486baae8100f7ee473f17e5-230
idx/v1-logtopics.0-64.ef                           1d90164285751a060dd682ead5c22e95-2
idx/v1-logtopics.64-80.ef                          2419dd716a988878735908108318d678-177
idx/v1-storage.0-64.ef                             94764763dad558e41330eddbba73f694-4
idx/v1-storage.64-72.ef                            0cdfeb0b92c34ff67bb94c86aa79677e-426
idx/v1-storage.64-80.ef                            3f35691eb594d738eb7f7d729c258c60-854
idx/v1-storage.72-76.ef                            07bebe6ebf7564957848acd7cbe590d6-247
idx/v1-storage.76-78.ef                            93267211ea04355ef3c3e7d255e3ec1e-143
idx/v1-storage.78-79.ef                            dc850879cddbd7683f1b706b1e77e67f-64
idx/v1-storage.79-80.ef                            89057241c789d896e37ac11df17c852f-65
idx/v1-storage.80-82.ef                            8f8b2038fc2332b5d1fd3fdbe59b6d13-124
idx/v1-tracesfrom.0-64.ef                          14767b774c9fe16143ccc158c63231c6-352
idx/v1-tracesto.0-64.ef                            eaa6273c9d3f661a12422248bc1f827d-417
v1-001500-002000-transactions.seg                  374d84bab0deba57cfc6539785b24741-85
v1-002000-002500-transactions.seg                  0a4a059adc5fb3bd42cc4613741512b0-101
v1-002500-003000-transactions.seg                  7758c576ec3da6cbe01a8705c3b3caca-101
v1-003000-003500-transactions.seg                  43a48f09307287b5d38492d0465455ff-1415
v1-003500-004000-transactions.seg                  92e1aa02e53d9924c56f000ab3ed79be-5842
v1-004000-004100-transactions.seg                  7042f47c8b8f390135fa6be2d9a397ab-3
v1-004100-004200-transactions.seg                  72596cdcdca0cced80282abeeea5346b-1946
v1-004200-004300-transactions.seg                  1e90c1d899425923ee82950ae97ab5d3-1522
v1-004300-004400-transactions.seg                  6cde71e8c8b2fa80e32350a5efe92c4e-2025
v1-004400-004500-transactions.seg                  713e339d74a91946d8b1bbe23976589d-1567
v1-004500-004600-transactions.seg                  e3e0b63d6a9bf3db4880ce925c61ec6a-1113
v1-004600-004700-transactions.seg                  8e3a2c4e6c62380c327826e34d06bdfe-1135
v1-004700-004800-transactions.seg                  c5b58b7f416f32ae295968cc6d4ea438-1245
v1-004800-004900-transactions.seg                  56177e381fb19ddeee961089e0620f69-1634
v1-004900-005000-transactions.seg                  07fada8ca9427297922bc90cf95b197a-1706

# ETag fetch failed
accessors/v1-accounts.0-64.efi
accessors/v1-accounts.0-64.vi
accessors/v1-accounts.64-72.efi
accessors/v1-accounts.64-72.vi
accessors/v1-accounts.64-80.efi
accessors/v1-accounts.64-80.vi
accessors/v1-accounts.72-76.efi
accessors/v1-accounts.72-76.vi
accessors/v1-accounts.76-78.efi
accessors/v1-accounts.76-78.vi
accessors/v1-accounts.78-79.efi
accessors/v1-accounts.78-79.vi
accessors/v1-accounts.79-80.efi
accessors/v1-accounts.79-80.vi
accessors/v1-accounts.80-82.efi
accessors/v1-accounts.80-82.vi
accessors/v1-code.0-64.efi
accessors/v1-code.0-64.vi
accessors/v1-code.64-80.efi
accessors/v1-code.64-80.vi
accessors/v1-code.80-82.efi
accessors/v1-code.80-82.vi
accessors/v1-logaddrs.0-64.efi
accessors/v1-logaddrs.64-80.efi
accessors/v1-logaddrs.80-82.efi
accessors/v1-logtopics.0-64.efi
accessors/v1-logtopics.64-80.efi
accessors/v1-logtopics.80-82.efi
accessors/v1-storage.0-64.efi
accessors/v1-storage.0-64.vi
accessors/v1-storage.64-72.efi
accessors/v1-storage.64-72.vi
accessors/v1-storage.64-80.efi
accessors/v1-storage.64-80.vi
accessors/v1-storage.72-76.efi
accessors/v1-storage.72-76.vi
accessors/v1-storage.76-78.efi
accessors/v1-storage.76-78.vi
accessors/v1-storage.78-79.efi
accessors/v1-storage.78-79.vi
accessors/v1-storage.79-80.efi
accessors/v1-storage.79-80.vi
accessors/v1-storage.80-82.efi
accessors/v1-storage.80-82.vi
accessors/v1-tracesfrom.0-64.efi
accessors/v1-tracesfrom.64-80.efi
accessors/v1-tracesfrom.80-82.efi
accessors/v1-tracesto.0-64.efi
accessors/v1-tracesto.64-80.efi
accessors/v1-tracesto.80-82.efi
== BAD https://erigon3-v3-snapshots-sepolia.erigon.networkn
EROR[03-21|14:13:26.996] webseed.http: status=404, url=https://erigon3-v1-snapshots-s

@awskii
Copy link
Member Author

awskii commented Mar 21, 2024

  • Do we seed /accessor directory?
  • so we do not create torrent for each file. Besides .txt, which files are also seeded without torrents?
  • .idx files seeded without torrents?

torrent_hash/ torrent_create/ manifest should produce same list of files (or print their names) at least i expected that.

For rclone need to add flags --s3-use-multipart-uploads=true --s3-use-multipart-etag=true --files-from $DATADIR/snapshots/manifest.txt

@awskii awskii changed the title added command to verify local files against webseed and manifest added command to verify remote manifests from webseeds Mar 21, 2024
@awskii
Copy link
Member Author

awskii commented Mar 21, 2024

Removed any verification that remote and local have same files, checking of md5 (only check if md5 have correct format) since it's done by rclone.

@awskii
Copy link
Member Author

awskii commented Mar 22, 2024

I think other thing is good to do is make a one-command alias for torrent clean+create+hashes + manifest creation.

logger.Info("[snapshots.webseed] processing local file etags")
removedTorrents := 0
walker := func(path string, de fs.DirEntry, err error) error {
if err != nil || de.IsDir() {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think you can get de nil ptr here - because err is not nil

@AskAlexSharov AskAlexSharov merged commit fa40ef6 into devel Mar 24, 2024
6 checks passed
@AskAlexSharov AskAlexSharov deleted the dev_manifest_verify branch March 24, 2024 10:33
@ghost ghost mentioned this pull request Jun 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants