[execution] Update to deps-only mode #20102
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
3 Skipped Deployments
|
tnowacki
reviewed
Oct 30, 2024
| async fn test_upgrade_package_add_new_module_in_dep_only_mode() { | ||
| let mut runner = UpgradeStateRunner::new("move_upgrade/base").await; | ||
|
|
||
| let (digest, modules) = build_upgrade_test_modules("dep_only_upgrade_new_module"); |
There was a problem hiding this comment.
would it be possible to slice out the new module here, retry the publish, and see that it succeeds?
There was a problem hiding this comment.
We have that exact test pretty much here: https://github.com/MystenLabs/sui/blob/main/crates/sui-core/src/unit_tests/move_package_upgrade_tests.rs#L573
This is the same package but without new_module.move.
There was a problem hiding this comment.
That is a slightly different from what I am asking. As it stands, it is not guaranteed that the error in this test from the fact that there is a new module.
If you take the modules list and splice it out, you would guarantee that within this test.
There was a problem hiding this comment.
Updated so they all start from the valid dep_only_upgrade (which is verified in each unit test), and then files are explicitly added or removed from that.
sui-execution/latest/sui-adapter/src/programmable_transactions/execution.rs
Outdated
Show resolved
Hide resolved
tzakian
force-pushed
the
tzakian/deps-only-no-new
branch
from
1ab5353 to
b43de36
Compare
tnowacki
approved these changes
Oct 30, 2024
carlosbmamoru
pushed a commit
to Mamoru-Foundation/sui
that referenced
this pull request
Nov 8, 2024
commit 7839b9501066108cb2322ba9039120a41781a1b0
Author: John Martin <johnjosephmartin@icloud.com>
Date: Mon Nov 4 14:39:56 2024 -0800
add client-request-method to CORS allowed headers (#20160) (#20168)
## Description
adds `client-request-method` to allowed cors methods
## Test plan
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
## Description
Describe the changes or additions included in this PR.
## Test plan
How did you test the new or updated feature?
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit ecffa454e86b994b79dfc6fb9032d02ee95a0697
Author: John Martin <johnjosephmartin@icloud.com>
Date: Mon Nov 4 13:13:22 2024 -0800
fix: remove high cardinality address label from safe_client_latency (… (#20169)
…#20165)
## Description
safe_client_latency is the highest cardinality metric across our
observability system due to the `address` label + being a histogram:
```
root@sui-node-mysten-rpc-0:/sui# curl -s localhost:9184/metrics | grep safe_client_latency_bucket | wc -l
8512
```
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
## Description
Describe the changes or additions included in this PR.
## Test plan
How did you test the new or updated feature?
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 1493483238672e1d5c22dbff15eb670b1d57883a
Author: Eugene Boguslavsky <eugene@mystenlabs.com>
Date: Mon Nov 4 20:00:02 2024 +0000
Sui v1.37.1 Version Bump
commit 6646b75dd68808b7bedb679c41706b12ae79a5a9
Author: Eugene Boguslavsky <eugene@mystenlabs.com>
Date: Thu Oct 31 18:46:36 2024 -0700
Sui `v1.37.0` Framework Bytecode Snapshot (#20131)
## Description
Sui `v1.37.0` Framework Bytecode Snapshot
## Test plan
`cargo run --bin sui-framework-snapshot`
commit fa5d48fcb3e06a11677dc14e90f12cac59e8f2fc
Author: Ashok Menon <ashok@mystenlabs.com>
Date: Fri Nov 1 01:03:44 2024 +0000
chore(indexer-alt): alphabetize Cargo.toml
commit e116dfa6547e96608bee6767a483e080bb0a1e99
Author: Ashok Menon <ashok@mystenlabs.com>
Date: Fri Nov 1 00:39:20 2024 +0000
indexer-alt: watermark timestamps and other logging improvements (#20120)
## Description
Track the timestamp associated with a watermark, include it in tracing
messages, and introduce gauges to track the timestamps of the latest
checkpoint gathered and written to the DB.
This change also introduces a couple of other timestamp/watermark
related tracing improvements:
- Measure watermarks gathered and in DB for the sequential pipeline
(previously just tracked the watermark after they were written to the
DB). These will usually be the exact same, but may differ if the
pipeline needs to retry a write.
- Gets rid of the ordering and equality impls for `CommitterWatermark`
-- these are no longer required.
- Standardise the ordering of watermark traces between the sequential
and concurrent pipelines, so that they are easier to compare with each
other (they now share a common prefix).
- Standardised tracing messages during committer/watermark teardown: Now
in each exit edge, we log the reason for the exit, and then a message
with the final watermark.
## Test plan
Run `sum_obj_types` and `wal_obj_types` with a consistent range of
`3600` and note that the difference in their timestamps is roughly an
hour (to begin with, it's about 70 minutes):
```
sui$ cargo run -p sui-indexer-alt --release -- \
--database-url 'postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt' \
indexer --remote-store-url 'https://checkpoints.mainnet.sui.io' \
--consistent-range 3600 --pipeline wal_obj_types --pipeline sum_obj_types`
Running `/Users/ashokmenon/sui/idx-poc/target/release/sui-indexer-alt --database-url 'postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt' indexer --remote-store-url 'https://checkpoints.mainnet.sui.io' --consistent-range 3600 --pipeline wal_obj_types --pipeline sum_obj_types`
2024-10-31T15:16:26.867221Z INFO sui_indexer_alt::db: Running migrations ...
2024-10-31T15:16:26.889145Z INFO sui_indexer_alt::db: Migrations complete.
2024-10-31T15:16:27.024406Z INFO sui_indexer_alt: Skipping pipeline ev_emit_mod
2024-10-31T15:16:27.024413Z INFO sui_indexer_alt: Skipping pipeline ev_struct_inst
2024-10-31T15:16:27.024414Z INFO sui_indexer_alt: Skipping pipeline kv_checkpoints
2024-10-31T15:16:27.024414Z INFO sui_indexer_alt: Skipping pipeline kv_objects
2024-10-31T15:16:27.024415Z INFO sui_indexer_alt: Skipping pipeline kv_transactions
2024-10-31T15:16:27.024416Z INFO sui_indexer_alt: Skipping pipeline obj_versions
2024-10-31T15:16:27.024416Z INFO sui_indexer_alt: Skipping pipeline tx_affected_objects
2024-10-31T15:16:27.024417Z INFO sui_indexer_alt: Skipping pipeline tx_balance_changes
2024-10-31T15:16:27.024417Z INFO sui_indexer_alt: Skipping pipeline wal_coin_balances
2024-10-31T15:16:27.033550Z INFO sui_indexer_alt: Skipping pipeline sum_coin_balances
2024-10-31T15:16:27.033607Z INFO sui_indexer_alt::pipeline::processor: Starting processor pipeline="wal_obj_types"
2024-10-31T15:16:27.033856Z INFO sui_indexer_alt::pipeline::concurrent::collector: Starting collector pipeline="wal_obj_types"
2024-10-31T15:16:27.033869Z INFO sui_indexer_alt::pipeline::concurrent::committer: Starting committer pipeline="wal_obj_types"
2024-10-31T15:16:27.033983Z INFO sui_indexer_alt: Skipping pipeline sum_packages
2024-10-31T15:16:27.034011Z INFO sui_indexer_alt::pipeline::processor: Starting processor pipeline="sum_obj_types"
2024-10-31T15:16:27.034183Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Starting watermark pipeline="wal_obj_types" watermark=CommitterWatermark { pipeline: "wal_obj_types", epoch_hi_inclusive: 0, checkpoint_hi_inclusive: 0, tx_hi: 0, timestamp_ms_hi_inclusive: 0 }
2024-10-31T15:16:27.034260Z INFO sui_indexer_alt::pipeline::sequential::committer: Starting committer pipeline="sum_obj_types" watermark=CommitterWatermark { pipeline: "sum_obj_types", epoch_hi_inclusive: 0, checkpoint_hi_inclusive: 0, tx_hi: 0, timestamp_ms_hi_inclusive: 0 }
2024-10-31T15:16:27.034305Z INFO sui_indexer_alt: Ingestion range first_checkpoint=0 last_checkpoint=None
2024-10-31T15:16:27.034339Z INFO sui_indexer_alt::metrics: Starting metrics service on 0.0.0.0:9184
2024-10-31T15:16:27.034351Z INFO sui_indexer_alt::ingestion::regulator: Starting ingestion regulator
2024-10-31T15:16:27.034355Z INFO sui_indexer_alt::ingestion::broadcaster: Starting ingestion broadcaster
2024-10-31T15:16:27.541400Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=212 transaction=213 timestamp=2023-04-13 13:28:25.836 UTC updated=true elapsed_ms=5.486208
2024-10-31T15:16:28.043533Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=515 transaction=516 timestamp=2023-04-13 13:36:17.240 UTC updated=true elapsed_ms=5.512666
2024-10-31T15:16:28.538421Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=1017 transaction=1018 timestamp=2023-04-13 13:48:04.584 UTC updated=true elapsed_ms=3.468209
2024-10-31T15:16:29.041788Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=1422 transaction=1423 timestamp=2023-04-13 13:57:11.577 UTC updated=true elapsed_ms=7.184292
2024-10-31T15:16:29.540675Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=1824 transaction=1825 timestamp=2023-04-13 14:05:59.154 UTC updated=true elapsed_ms=5.7144580000000005
2024-10-31T15:16:30.040987Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=2226 transaction=2227 timestamp=2023-04-13 14:14:54.721 UTC updated=true elapsed_ms=5.955416
2024-10-31T15:16:30.539394Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=2630 transaction=2631 timestamp=2023-04-13 14:23:21.636 UTC updated=true elapsed_ms=5.0003340000000005
2024-10-31T15:16:31.035944Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=3142 transaction=3143 timestamp=2023-04-13 14:34:00.192 UTC updated=true elapsed_ms=1.368042
2024-10-31T15:16:31.532346Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=71 transaction=72 timestamp=2023-04-13 13:24:36.662 UTC
2024-10-31T15:16:31.535237Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=3646 transaction=3647 timestamp=2023-04-13 14:45:19.227 UTC updated=true elapsed_ms=0.7080409999999999
2024-10-31T15:16:31.556459Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=103 transaction=104 timestamp=2023-04-13 13:25:35.491 UTC
2024-10-31T15:16:31.605247Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=151 transaction=152 timestamp=2023-04-13 13:27:01.087 UTC
2024-10-31T15:16:31.658676Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=203 transaction=204 timestamp=2023-04-13 13:28:14.431 UTC
2024-10-31T15:16:31.709093Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=251 transaction=252 timestamp=2023-04-13 13:29:41.350 UTC
2024-10-31T15:16:31.774101Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=304 transaction=305 timestamp=2023-04-13 13:31:05.962 UTC
2024-10-31T15:16:31.823847Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=351 transaction=352 timestamp=2023-04-13 13:32:06.748 UTC
2024-10-31T15:16:31.875613Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=404 transaction=405 timestamp=2023-04-13 13:33:26.122 UTC
2024-10-31T15:16:31.923673Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=452 transaction=453 timestamp=2023-04-13 13:34:41.672 UTC
2024-10-31T15:16:31.973523Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=502 transaction=503 timestamp=2023-04-13 13:35:54.245 UTC
2024-10-31T15:16:32.020968Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=553 transaction=554 timestamp=2023-04-13 13:37:12.723 UTC
2024-10-31T15:16:32.034336Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=4147 transaction=4148 timestamp=2023-04-13 14:56:30.470 UTC updated=true elapsed_ms=0.478916
2024-10-31T15:16:32.081115Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=603 transaction=604 timestamp=2023-04-13 13:38:26.226 UTC
2024-10-31T15:16:32.143228Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=657 transaction=658 timestamp=2023-04-13 13:39:32.583 UTC
2024-10-31T15:16:32.201323Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=710 transaction=711 timestamp=2023-04-13 13:40:44.461 UTC
2024-10-31T15:16:32.239715Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=751 transaction=752 timestamp=2023-04-13 13:41:40.955 UTC
2024-10-31T15:16:32.306828Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=803 transaction=804 timestamp=2023-04-13 13:43:00.433 UTC
2024-10-31T15:16:32.351995Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=854 transaction=855 timestamp=2023-04-13 13:44:21.487 UTC
2024-10-31T15:16:32.408044Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=903 transaction=904 timestamp=2023-04-13 13:45:33.468 UTC
2024-10-31T15:16:32.478623Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=963 transaction=964 timestamp=2023-04-13 13:46:48.531 UTC
2024-10-31T15:16:32.517822Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1004 transaction=1005 timestamp=2023-04-13 13:47:43.997 UTC
2024-10-31T15:16:32.540701Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=4550 transaction=4551 timestamp=2023-04-13 15:05:30.690 UTC updated=true elapsed_ms=5.776375
2024-10-31T15:16:32.574600Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1052 transaction=1053 timestamp=2023-04-13 13:48:55.253 UTC
2024-10-31T15:16:32.625726Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1102 transaction=1103 timestamp=2023-04-13 13:49:57.288 UTC
2024-10-31T15:16:32.679135Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1159 transaction=1160 timestamp=2023-04-13 13:51:06.819 UTC
2024-10-31T15:16:32.729858Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1203 transaction=1204 timestamp=2023-04-13 13:52:11.201 UTC
2024-10-31T15:16:32.777044Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1257 transaction=1258 timestamp=2023-04-13 13:53:26.750 UTC
2024-10-31T15:16:32.819603Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1301 transaction=1302 timestamp=2023-04-13 13:54:17.123 UTC
2024-10-31T15:16:32.870028Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1351 transaction=1352 timestamp=2023-04-13 13:55:24.124 UTC
2024-10-31T15:16:32.923059Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1402 transaction=1403 timestamp=2023-04-13 13:56:49.078 UTC
2024-10-31T15:16:32.969159Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1451 transaction=1452 timestamp=2023-04-13 13:57:56.351 UTC
2024-10-31T15:16:33.034831Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=5055 transaction=5056 timestamp=2023-04-13 15:16:38.038 UTC updated=true elapsed_ms=1.0442500000000001
2024-10-31T15:16:33.036266Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1501 transaction=1502 timestamp=2023-04-13 13:58:56.227 UTC
^C2024-10-31T15:16:33.091189Z INFO sui_indexer_alt::pipeline::concurrent::collector: Shutdown received, stopping collector pipeline="wal_obj_types"
2024-10-31T15:16:33.091195Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Shutdown received pipeline="wal_obj_types"
2024-10-31T15:16:33.091201Z INFO sui_indexer_alt::ingestion::regulator: Shutdown received, stopping regulator
2024-10-31T15:16:33.091213Z INFO sui_indexer_alt::metrics: Shutdown received, stopping metrics service
2024-10-31T15:16:33.091225Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Stopping watermark task pipeline="wal_obj_types" watermark=CommitterWatermark { pipeline: "wal_obj_types", epoch_hi_inclusive: 0, checkpoint_hi_inclusive: 5055, tx_hi: 5056, timestamp_ms_hi_inclusive: 1681398998038 }
2024-10-31T15:16:33.091410Z INFO sui_indexer_alt::pipeline::concurrent::committer: Batches done, stopping committer pipeline="wal_obj_types"
2024-10-31T15:16:33.091721Z INFO sui_indexer_alt::pipeline::sequential::committer: Shutdown received pipeline="sum_obj_types"
2024-10-31T15:16:33.091725Z INFO sui_indexer_alt::pipeline::sequential::committer: Stopping committer pipeline="sum_obj_types" watermark=CommitterWatermark { pipeline: "sum_obj_types", epoch_hi_inclusive: 0, checkpoint_hi_inclusive: 1546, tx_hi: 1547, timestamp_ms_hi_inclusive: 1681394391225 }
2024-10-31T15:16:33.093829Z INFO sui_indexer_alt::pipeline::processor: Shutdown received, stopping processor pipeline="wal_obj_types"
2024-10-31T15:16:33.094014Z INFO sui_indexer_alt::pipeline::processor: Shutdown received, stopping processor pipeline="sum_obj_types"
2024-10-31T15:16:33.094406Z INFO sui_indexer_alt::ingestion::broadcaster: Shutdown received, stopping ingestion broadcaster
2024-10-31T15:16:33.094513Z INFO sui_indexer_alt: Indexing pipeline gracefully shut down
```
## Stack
- #20089
- #20114
- #20116
- #20117
- #20119
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 4b727cbbfd89db7ebfd09b21e05faa58ac2d8c96
Author: Ashok Menon <ashok@mystenlabs.com>
Date: Fri Nov 1 00:37:19 2024 +0000
indexer-alt: allow sequential pipeline immediate reset (#20119)
## Description
If the sequential pipeline committer can guarantee that it could process
more checkpoints by looking at its pending buffer, we now reset the
polling interval immediately, so it does not wait to issue the next
write. This mimics a similar behaviour in the concurrent pipeline.
I made this change after noticing how the pipeline behaves when
ingestion is stuck retrying a checkpoint, on my local machine. Usually
when running locally, performance is limited by checkpoint download
rate, but in a sequential pipeline, if a checkpoint failed to download,
it is possible for many checkpoints to end up processed and pending.
With the previous implementation, once ingestion had recovered (the
checkpoint is fetched), the pending buffer kept growing because it was
only able to land `MAX_BATCH_CHECKPOINTS / commit_interval`, so if
checkpoints were getting added faster than that, it would never recover.
With this change, the pipeline recovers almost instantly, and I expect
that in GCP where bandwidth is not the rate limiting factor, this should
improve throughput during backfill, and synthetic benchmarks.
## Test plan
Run the indexer with a large ingestion buffer and concurrency, wait for
ingestion to fail to fetch a checkpoint, and then notice how the
situation recovers (instead of getting worse until the pipeline
eventually complains that it has too many pending checkpoints):
```
sui$ cargo run -p sui-indexer-alt --release -- \
--database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \
indexer --remote-store-url https://checkpoints.mainnet.sui.io \
--last-checkpoint 1200000 --pipeline sum_packages \
--checkpoint-buffer-size 50000 --ingest-concurrency 20000
```
## Stack
- #20089
- #20114
- #20116
- #20117
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit b65f34c6a00801620e11469575523895a8e074ac
Author: Ashok Menon <ashok@mystenlabs.com>
Date: Fri Nov 1 00:21:38 2024 +0000
indexer-alt: wal_coin_balances pipeline (#20117)
## Description
`wal_coin_balances` is to `sum_coin_balances` what `wal_obj_types` is to
`sum_obj_types`.
## Test plan
Run the indexer, and correlate the live object set calculated from the
write-ahead log against the one that's already in the summary table:
```
sui$ cargo run -p sui-indexer-alt --release -- \
--database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \
indexer --remote-store-url https://checkpoints.mainnet.sui.io/ \
--last-checkpoint 5000
```
```
sui_indexer_alt=# SELECT COUNT(*) FROM sum_coin_balances;
count
-------
178
(1 row)
sui_indexer_alt=# SELECT
COUNT(*)
FROM (
SELECT DISTINCT ON (object_id)
*
FROM
wal_coin_balances
ORDER BY
object_id,
object_version DESC
) o
WHERE
o.owner_id IS NOT NULL;
count
-------
178
(1 row)
```
## Stack
- #20089
- #20114
- #20116
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 71f1faa1a8557a178e0bf89f20829a8ea503f156
Author: Ashok Menon <ashok@mystenlabs.com>
Date: Fri Nov 1 00:18:46 2024 +0000
indexer-alt: wal_obj_types pipeline (#20116)
## Description
Adds the concurrent pipeline that writes the write-ahead log for
`sum_obj_types`. It re-uses the `process` implementation from
`sum_obj_types` and then writes it into an append-only table.
Note that today:
- The pipelines are completely independent, which means the WAL pipeline
redoes the processing work of the summary pipeline (this is presumably
not an issue because the repeated work is not particularly heavy).
- This change does not include the pruning necessary to keep this
table's size in check (in practice it should only be a couple of gigs in
size). This will come in a follow-up PR.
## Test plan
Run the indexer and cross check the live-object set calculated from the
write-ahead log with the summary:
```
sui$ cargo run -p sui-indexer-alt --release -- \
--database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \
indexer --remote-store-url https://checkpoints.mainnet.sui.io \
--last-checkpoint 5000
```
```
sui_indexer_alt=# SELECT COUNT(*) FROM sum_obj_types;
count
-------
592
(1 row)
^
sui_indexer_alt=# SELECT
COUNT(*)
FROM (
SELECT DISTINCT ON (object_id)
*
FROM
wal_obj_types
ORDER BY
object_id,
object_version DESC
) o
WHERE
o.owner_kind IS NOT NULL;
count
-------
592
(1 row)
```
## Stack
- #20089
- #20114
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 7de02417c27844c8a086866b1279b4736ab862ec
Author: Ashok Menon <ashok@mystenlabs.com>
Date: Thu Oct 31 23:22:42 2024 +0000
indexer-alt: obj_versions pipeline (#20114)
## Description
Pipeline to fill `obj_versions` table -- this is the table that serves
"parent version" (dynamic field) queries.
## Test plan
```
sui$ cargo run -p sui-indexer-alt --release -- \
--database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \
indexer --remote-url "https://checkpoints.mainnet.sui.io" \
--last-checkpoint 5000
```
## Stack
- #20089
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 53eaed5e300a67f0d01cf68119a145410925f9de
Author: Ashok Menon <ashok@mystenlabs.com>
Date: Wed Oct 30 00:06:07 2024 +0000
indexer-alt: delays for sequential pipelines
## Description
Add the ability to include an optional delay to sequential pipelines.
This can be used to implement behaviour similar to `objects_snapshot` in
the new architecture.
Sequential pipelines honour this configuration in their committer tasks:
- They process checkpoints at the same rate as other pipelines
- But hold back data in the committer's "pending" map until the
committer witnesses a checkpoint that is at least `checkpoint_lag`
checkpoints ahead.
This also requires slightly complicating the set-up and tear-down logic:
- When setting up, we may need to wait until the pending buffer has been
primed enough to set a valid commit upperbound.
- If the pipeline runs dry before that point, we need to add an extra
check to close down the committer.
- When tearing down the committer after its channel has closed, we need
to check whether batching would have picked up the first pending
checkpoint (whether it is a valid "next checkpoint" and it's not held
up because of a lag parameter).
## Test plan
Run the indexer with various parameterisations for the consistent range,
and note that the watermark for the summary pipelines lag behind the
watermarks for other pipelines.
```
sui$ cargo run -p sui-indexer-alt --release -- \
--database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \
indexer --remote-store-url https://checkpoints.mainnet.sui.io \
--pipeline kv_objects \
--pipeline sum_obj_types --pipeline sum_coin_balances \
--last-checkpoint 5000 --consistent-range 1000
[...]
2024-10-30T00:24:45.148259Z INFO sui_indexer_alt::ingestion::regulator: Checkpoints done, stopping regulator
2024-10-30T00:24:45.161682Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=3605 transaction=3606
2024-10-30T00:24:45.212394Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_coin_balances" epoch=0 checkpoint=2703 transaction=2704
2024-10-30T00:24:45.222986Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=3656 transaction=3657
2024-10-30T00:24:45.274318Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=3702 transaction=3703
2024-10-30T00:24:45.340714Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=3753 transaction=3754
2024-10-30T00:24:45.355934Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="kv_objects" elapsed_ms=0.949083 updated=true epoch=0 checkpoint=4760 transaction=4761
2024-10-30T00:24:45.401262Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=3801 transaction=3802
2024-10-30T00:24:45.463134Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=3852 transaction=3853
2024-10-30T00:24:45.544933Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=3904 transaction=3905
2024-10-30T00:24:45.609068Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=3952 transaction=3953
2024-10-30T00:24:45.668560Z INFO sui_indexer_alt::ingestion::broadcaster: Checkpoints done, stopping ingestion broadcaster
2024-10-30T00:24:45.668593Z INFO sui_indexer_alt::pipeline::processor: Checkpoints done, stopping processor pipeline="kv_objects"
2024-10-30T00:24:45.668605Z INFO sui_indexer_alt::pipeline::processor: Checkpoints done, stopping processor pipeline="sum_obj_types"
2024-10-30T00:24:45.668605Z INFO sui_indexer_alt::pipeline::processor: Checkpoints done, stopping processor pipeline="sum_coin_balances"
2024-10-30T00:24:45.674305Z INFO sui_indexer_alt::pipeline::sequential::committer: Processor closed channel, pending rows empty, stopping committer pipeline="sum_obj_types" watermark=CommitterWatermark { pipeline: "sum_obj_types", epoch_hi_inclusive: 0, checkpoint_hi_inclusive: 4000, tx_hi: 4001 }
2024-10-30T00:24:45.713856Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_coin_balances" epoch=0 checkpoint=3003 transaction=3004
2024-10-30T00:24:45.863837Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="kv_objects" elapsed_ms=1.83825 updated=true epoch=0 checkpoint=4960 transaction=4961
2024-10-30T00:24:46.120459Z INFO sui_indexer_alt::pipeline::concurrent::collector: Processor closed channel, pending rows empty, stopping collector pipeline="kv_objects"
2024-10-30T00:24:46.131407Z INFO sui_indexer_alt::pipeline::concurrent::committer: Batches done, stopping committer pipeline="kv_objects"
2024-10-30T00:24:46.214267Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_coin_balances" epoch=0 checkpoint=3303 transaction=3304
2024-10-30T00:24:46.356941Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="kv_objects" elapsed_ms=0.68 updated=true epoch=0 checkpoint=5000 transaction=5001
2024-10-30T00:24:46.356970Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Committer closed channel, stopping watermark task pipeline="kv_objects" watermark=CommitterWatermark { pipeline: "kv_objects", epoch_hi_inclusive: 0, checkpoint_hi_inclusive: 5000, tx_hi: 5001 }
2024-10-30T00:24:46.714498Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_coin_balances" epoch=0 checkpoint=3603 transaction=3604
2024-10-30T00:24:47.214758Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_coin_balances" epoch=0 checkpoint=3903 transaction=3904
2024-10-30T00:24:47.715841Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_coin_balances" epoch=0 checkpoint=4000 transaction=4001
2024-10-30T00:24:47.715917Z INFO sui_indexer_alt::pipeline::sequential::committer: Processor closed channel, pending rows empty, stopping committer pipeline="sum_coin_balances" watermark=CommitterWatermark { pipeline: "sum_coin_balances", epoch_hi_inclusive: 0, checkpoint_hi_inclusive: 4000, tx_hi: 4001 }
2024-10-30T00:24:47.716485Z INFO sui_indexer_alt: Indexing pipeline gracefully shut down
2024-10-30T00:24:47.716647Z INFO sui_indexer_alt::metrics: Shutdown received, stopping metrics service
```
commit d57326492b62ec8ba7ff8826ad473ec1b5008f11
Author: Ashok Menon <ashok@mystenlabs.com>
Date: Wed Oct 30 00:01:45 2024 +0000
fix(indexer-alt): remove double logging on pipeline skip
## Description
We were logging the pipeline being skipped twice.
## Test plan
Check logs while the indexer starts up:
```
sui$ cargo run -p sui-indexer-alt --release -- \
--database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \
indexer --remote-store-url "https://checkpoints.mainnet.sui.io" \
--last-checkpoint 5000 --pipeline sum_obj_types
```
commit ef0d78c638e3ef0d0a3b28787b618ea56a140284
Author: Ge Gao <106119108+gegaowp@users.noreply.github.com>
Date: Thu Oct 31 19:30:51 2024 -0400
add sui-field-count for sui-indexer-alt (#20126)
## Description
title, so that we can replace hard-coded field count in pipelines.
a crate did very similar things but it's archived, so prob better to
have our own impl, also b/c the functionality is relatively simple
https://github.com/discosultan/field-count
## Test plan
added tests in models/
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit c92dd1b4f16074ced8c1fe5c56a10e675f68bbe6
Author: phoenix <51927076+phoenix-o@users.noreply.github.com>
Date: Thu Oct 31 18:28:34 2024 -0400
[kv store] bigtable. Add support for querying checkpoint by digest (#20122)
## Description
Adds support for querying checkpoint by digest
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 74ace7b6166f10bddfd714568e0ddc20db2ca75d
Author: JasonRUAN <rzexin@gmail.com>
Date: Fri Nov 1 06:25:44 2024 +0800
[CLI] fix `sui keytool import` --alias command and default alias does not work (#20111)
## Description
fix `sui keytool import` --alias command and default alias does not work
Before Fix：
<img width="1295" alt="image"
src="https://github.com/user-attachments/assets/4cb40d92-e266-4b6e-9c39-c05ac52d7a9d">
After Fix：
- **--alias**
<img width="1301" alt="image"
src="https://github.com/user-attachments/assets/b5c711e8-2a4d-4861-9965-30cde260e93f">
- **default alias**
<img width="1295" alt="image"
src="https://github.com/user-attachments/assets/4513db65-0e10-4179-b5f0-de188871cf78">
## Test plan
How did you test the new or updated feature?
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [x] CLI: `sui keytool import --alias` is now fixed and correctly
records the provided alias.
- [ ] Rust SDK:
- [ ] REST API:
commit e41a759769fdea08f660160a23e945bbad80b337
Author: sui-merge-bot[bot] <114704316+sui-merge-bot[bot]@users.noreply.github.com>
Date: Thu Oct 31 23:50:37 2024 +0200
Version Packages (#20127)
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.
# Releases
## @mysten/kms@0.0.3
### Patch Changes
- 02c9e46: Fix exports on the bundled package
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
commit 02c9e4691b429b4a0ec85b71eace37b72515d055
Author: Manolis Liolios <manos@mystenlabs.com>
Date: Thu Oct 31 23:38:15 2024 +0200
Fix build, re-publish @mysten/kms (#20125)
## Description
Build will always treat `src` as the base for outputs when building.
## Test plan
How did you test the new or updated feature?
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 16f2683a99e7b8824aa5879d3f20ffef936a7546
Author: William Smith <williamprincesmith@gmail.com>
Date: Thu Oct 31 17:27:16 2024 -0400
[sui-benchmark] Add expected failure payloads (#19831)
## Description
Introduce a new payload type into the benchmark tool -
`ExpectedFailurePayload`.
This payload type is configured with some expected failure type
(currently only one implemented - user signature failure - but others
can be added with minimal scaffolding) such that
`payload.make_transaction()` generates a transaction that will fail in
this manner. Note that the failures that this payload type is concerned
with are failures to execute the transaction itself, rather than during
execution by the MoveVM. In other words, it is expected that the failure
mode will not consume gas or produce effects.
Note that for this reason, its failure mode is inverted. It will be
tallied for metrics purposes as an `error` if it succeeds and tallied as
an `expected_error` (which is a success) if it fails.
Also note that transaction responses for this type are handled by
producing a `NextOp::Retry` (with some additional logging/metrics) since
it is functionally equivalent to a retryable error.
## Test plan
Ran the following:
```
SIM_STRESS_TEST_QPS=200 SIM_STRESS_TEST_WORKERS=20 RUST_LOG=debug cargo simtest --nocapture test_simulated_load_expected_failure_traffic_control
```
Observed the following in logs
```
2022-01-03T02:05:47.734507Z INFO node{id=1 name="client"}: simtest::test: crates/sui-benchmark/tests/simtest.rs:1088: end of test BenchmarkStats { duration: 50.000002242s, num_error_txes: 0, num_expected_error_txes: 8049, num_success_txes: 348, num_success_cmds: 1977, total_gas_used: 3215776400, latency_ms: HistogramWrapper { histogram: Histogram { auto_resize: false, highest_trackable_value: 120000, lowest_discernible_value: 1, significant_value_digits: 3, bucket_count: 7, sub_bucket_count: 2048, sub_bucket_half_count: 1024, sub_bucket_half_count_magnitude: 10, sub_bucket_mask: 2047, leading_zero_count_base: 53, unit_magnitude: 0, unit_magnitude_mask: 0, max_value: 14359, min_non_zero_value: 72, total_count: 348, counts: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 8, 12, 9, 20, 25, 22, 19
...
```
[Also ran in
PTN](https://metrics.sui.io/d/adl51ctsvmkg0a/traffic-control-dos-protection-dashboard?from=2024-10-31T19:00:50.548Z&to=2024-10-31T20:20:03.673Z&timezone=browser&var-network=private-testnet&var-host=$__all&var-fullnode=$__all)
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit d95918cea0d8fa0f1a4007273f477e3c12b61302
Author: phoenix <51927076+phoenix-o@users.noreply.github.com>
Date: Thu Oct 31 17:18:17 2024 -0400
[data ingestion] add support for bigtable in main workflow binary (#20018)
## Description
adding BigTable KV as a variant to main binary that manages internal
workflows
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit c33374e735257b2a460a2f882e062619f8958bc5
Author: wlmyng <127570466+wlmyng@users.noreply.github.com>
Date: Thu Oct 31 12:51:47 2024 -0700
[mvr] indexer schema subset and reduced persist tasks (#20100)
## Description
Describe the changes or additions included in this PR.
## Test plan
How did you test the new or updated feature?
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 7c517e7e5ce7193d1f51ef7f20a31b56f0e66797
Author: Xun Li <lxfind@gmail.com>
Date: Thu Oct 31 12:49:58 2024 -0700
Add system_invariant_violation macro (#20105)
## Description
Introduce a system invariant violation macro.
We can then set up alerts whenever the counter value changes.
Used it in one place as an example.
## Test plan
How did you test the new or updated feature?
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit b248f35879801b3bae9b76c4d204f36540d816f8
Author: Andrew Schran <aschran@mystenlabs.com>
Date: Thu Oct 31 14:16:48 2024 -0400
Add support for TLS connections with self-signed cert on validator gRPC interface (#19796)
Client side use of TLS will be enabled-by-default in a future PR.
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [x] Nodes (Validators and Full nodes): Adds support for TLS on
validator gRPC interface.
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit e06610823b985bdc01c40e23c9a9a2f8d5b571d1
Author: sui-merge-bot[bot] <114704316+sui-merge-bot[bot]@users.noreply.github.com>
Date: Thu Oct 31 19:58:58 2024 +0200
Version Packages (#20076)
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.
# Releases
## @mysten/sui@1.14.0
### Minor Changes
- c24814b: Adds a custom header; 'Client-Request-Method' which will
contain the method name used in
each outgoing jsonrpc request
## @mysten/create-dapp@0.3.28
### Patch Changes
- Updated dependencies [c24814b]
- @mysten/sui@1.14.0
- @mysten/dapp-kit@0.14.28
## @mysten/dapp-kit@0.14.28
### Patch Changes
- Updated dependencies [c24814b]
- @mysten/sui@1.14.0
- @mysten/wallet-standard@0.13.9
- @mysten/zksend@0.11.9
## @mysten/deepbook@0.8.23
### Patch Changes
- Updated dependencies [c24814b]
- @mysten/sui@1.14.0
## @mysten/deepbook-v3@0.12.2
### Patch Changes
- Updated dependencies [c24814b]
- @mysten/sui@1.14.0
## @mysten/enoki@0.4.7
### Patch Changes
- Updated dependencies [c24814b]
- @mysten/sui@1.14.0
- @mysten/zklogin@0.7.24
## @mysten/graphql-transport@0.2.25
### Patch Changes
- Updated dependencies [c24814b]
- @mysten/sui@1.14.0
## @mysten/kiosk@0.9.23
### Patch Changes
- 4166d71: Fix doc comment on `getKiosk` command
- Updated dependencies [c24814b]
- @mysten/sui@1.14.0
## @mysten/kms@0.0.2
### Patch Changes
- b3f3925: Introduces @mysten/kms which initially exposes a Sui AWS KMS
signer
- Updated dependencies [c24814b]
- @mysten/sui@1.14.0
## @mysten/suins-toolkit@0.5.23
### Patch Changes
- Updated dependencies [c24814b]
- @mysten/sui@1.14.0
## @mysten/wallet-standard@0.13.9
### Patch Changes
- Updated dependencies [c24814b]
- @mysten/sui@1.14.0
## @mysten/zklogin@0.7.24
### Patch Changes
- Updated dependencies [c24814b]
- @mysten/sui@1.14.0
## @mysten/zksend@0.11.9
### Patch Changes
- Updated dependencies [c24814b]
- @mysten/sui@1.14.0
- @mysten/wallet-standard@0.13.9
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
commit 093954a09bce1942a9f9773b845f59bf5124efbf
Author: ronny-mysten <118224482+ronny-mysten@users.noreply.github.com>
Date: Thu Oct 31 11:35:27 2024 -0600
[docs] Rpc update (#20094)
## Description
Updates the RPC render to include component schemas. Also updates a few
comments.
## Test plan
How did you test the new or updated feature?
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit fa66617b7dfa7096fd64a2dc1b002d154223be4a
Author: wlmyng <127570466+wlmyng@users.noreply.github.com>
Date: Thu Oct 31 10:18:57 2024 -0700
[mvr] renaming to sui-mvr-indexer and Dockerfile (#20099)
## Description
Describe the changes or additions included in this PR.
## Test plan
How did you test the new or updated feature?
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 434206f14a0a21bc62ea08480b4a5c4d36d6cc25
Author: Todd Nowacki <tmn@mystenlabs.com>
Date: Thu Oct 31 10:18:46 2024 -0700
[move-compiler] Re-organize warning filters (#20103)
## Description
- Re-organize warning filters. It should make it a bit cleaner to
refactor diagnostic reporting
## Test plan
- ran tests
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 0d16b6f8b7c4ff79f70400e8a94d7486b2b2c3fe
Author: wlmyng <127570466+wlmyng@users.noreply.github.com>
Date: Thu Oct 31 09:36:37 2024 -0700
[mvr] new crate for custom mvr-indexer, copied from sui-indexer crate (#20098)
## Description
Simple copy-paste of the sui-indexer crate for the custom mvr indexer
## Test plan
How did you test the new or updated feature?
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit b3f39253ff0153b69b42bc298d1ecdbb0524e276
Author: John Naulty Jr. <jnaulty@mystenlabs.com>
Date: Thu Oct 31 09:21:29 2024 -0700
AWS KMS TS Signer (#16139)
## Description
Describe the changes or additions included in this PR.
## Test Plan
How did you test the new or updated feature?
---
If your changes are not user-facing and do not break anything, you can
skip the following section. Otherwise, please briefly describe what has
changed under the Release Notes section.
### Type of Change (Check all that apply)
- [ ] protocol change
- [ ] user-visible impact
- [ ] breaking change for a client SDKs
- [ ] breaking change for FNs (FN binary must upgrade)
- [ ] breaking change for validators or node operators (must upgrade
binaries)
- [ ] breaking change for on-chain data layout
- [ ] necessitate either a data wipe or data migration
### Release notes
---------
Co-authored-by: Manolis Liolios <manos@mystenlabs.com>
commit f78c7b006c28c4c1afd3f7d42e21e53970902f5d
Author: Brandon Williams <brandon@mystenlabs.com>
Date: Wed Oct 30 19:50:45 2024 -0500
rest: revert client change to requested protobuf and request bcs instead
Revert the client change to request protobuf as a response type, and
instead request bcs, because the support for protobuf hasn't rolled out
to other networks yet.
commit 108ad4c8ea53c68bd90d0b3fcdc88fbc75640c61
Author: Brandon Williams <brandon@mystenlabs.com>
Date: Wed Oct 30 12:50:43 2024 -0500
rest: support protobuf format for committee endpoints
commit 718a244bca584382faedde56f8247c6b26225b07
Author: Brandon Williams <brandon@mystenlabs.com>
Date: Wed Oct 30 11:03:43 2024 -0500
rest: support protobuf format for resolve, simulate, and execute transaction endpoints
commit 2987f5967ecffa8ddbec2b8ebf9ee2eec09dad60
Author: Brandon Williams <brandon@mystenlabs.com>
Date: Wed Oct 30 08:55:51 2024 -0500
rest: support protobuf format for GetTransaction and ListTransactions endpoints
commit 672ecbd2a58c7514aa24d139c442316a03d7f89e
Author: Brandon Williams <brandon@mystenlabs.com>
Date: Tue Oct 29 21:30:42 2024 -0500
rest: support protobuf format for GetObject endpoints
commit c8472c057523ff88040ae688f1410d34164b0ebc
Author: Brandon Williams <brandon@mystenlabs.com>
Date: Tue Oct 29 19:00:39 2024 -0500
rest: remove ListFullCheckpoint api
Remove the ListFullCheckpoint api as its a bit too heavy-weight and
instead we'll look to introduce a streaming api in the future.
commit 39f1ae21fd5467e71522729519720d2add9468dc
Author: Xun Li <lxfind@gmail.com>
Date: Wed Oct 30 22:25:50 2024 -0700
Clean up local execution (#20086)
## Description
Describe the changes or additions included in this PR.
## Test plan
How did you test the new or updated feature?
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 65b02e203ad897f438f87d0e5cfe46c56a1d4247
Author: phoenix <51927076+phoenix-o@users.noreply.github.com>
Date: Wed Oct 30 21:26:54 2024 -0400
[pruner] enable periodic compaction of individual SST files by default (#20101)
## Description
enables periodic compaction of individual SST files by default
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit f2670718f694edd59e8a062bc932d4efd96ba765
Author: Tim Zakian <2895723+tzakian@users.noreply.github.com>
Date: Wed Oct 30 16:49:01 2024 -0700
[execution] Update to deps-only mode (#20102)
## Description
Disallow new modules from being added to packages in deps-only mode.
## Test plan
Added new tests to check for both previous and new behavior.
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [X] Protocol: Updated restrictions on deps-only packages to not allow
adding new modules to the package.
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 7e6319fa33b7021a94ff7dcc07e479a4bae92b55
Author: Adam Welc <adam@mystenlabs.com>
Date: Wed Oct 30 15:51:37 2024 -0700
[trace-view] Added support for native functions and global locations (#20080)
## Description
Adds support for native functions where the `CloseFrame` event happens
right after `OpenFrame` event and both need to be skipped to skip over
the native function. Also added support for handling global locations -
they should be parsed properly but largely ignored in the trace as they
cannot directly affect values of local variables
## Test plan
All old and new tests must pass
commit 21639a51847f1a04b6b221ac77df3875c0430be7
Author: Xun Li <lxfind@gmail.com>
Date: Wed Oct 30 14:30:59 2024 -0700
[RPC] Fix coin metadata api inefficiency (#19794)
## Description
This PR fixes a minor inefficiency in the coin metadata api.
It does not need to fetch the transaction, as it only needs the effects.
This likely avoids a remote KV fetch.
## Test plan
CI
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit c24814b54853d70c42cfe17960043458a670e50d
Author: John Martin <johnjosephmartin@icloud.com>
Date: Wed Oct 30 13:22:18 2024 -0700
add Client-Request-Method header to all TS SDK requests (#20071)
## Description
This will be used by the new edge proxy for routing execute transaction
requests to fullnodes with lower consensus latency
## Test plan
unit tests
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 8f9c6fc2da199ef62d79f0e2bedf28ab30fb4a7c
Author: Tim Zakian <2895723+tzakian@users.noreply.github.com>
Date: Wed Oct 30 12:59:46 2024 -0700
[move] Add support for raw `abort`s in clever errors (#20063)
## Description
This adds support for raw `abort`s in a similar manner to raw
`assert!`ions. It will introduce the line number that the `abort` was
called on (and behaves in the same manner as `assert!` with no error
codes for macro line number remapping).
## Test plan
Added parser tests as well as additional transactional tests, and
verified that they produce the exact same output as the derived line
number assertion tests that they copy.
commit b261681d2135ebef9adeb6b09b89b8d0df12f695
Author: Tim Zakian <2895723+tzakian@users.noreply.github.com>
Date: Wed Oct 30 12:59:26 2024 -0700
[move] Unify `debugging` and `tracing` feature flags for tracing in the VM (#20084)
## Description
This unifies the old tracing under the `tracing` flag, and removes the
`debugging` feature flag from the VM (one less feature flag 🎉).
Otherwise behavior is kept the exact same.
## Test plan
Tested manually + CI
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit ebffb7540b78c68522bf465fe5f73198f780a490
Author: Eugene Boguslavsky <eugene@mystenlabs.com>
Date: Wed Oct 30 11:10:29 2024 -0700
Add emoji to slack error messages (#20092)
## Description
Add emoji to slack error messages
## Test plan
👀
commit 831e97a52d63b043f642882b31974db33a757911
Author: wlmyng <127570466+wlmyng@users.noreply.github.com>
Date: Wed Oct 30 11:04:26 2024 -0700
[indexer] drop tx_recipients and tx_senders tables from schema (#20087)
## Description
Describe the changes or additions included in this PR.
## Test plan
How did you test the new or updated feature?
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 419bac683f46da4a7eadad4617ea0679be493eeb
Author: Jort <jordan.jennings@mystenlabs.com>
Date: Wed Oct 30 11:02:33 2024 -0700
[CLI] upgrade error formatting should use existing diag wrapper (#20014)
## Description
Reuse the work within `move-compiler/src/diagnostics/` instead of
reimplementing code span reporting usage directly.
## Test…
carlosbmamoru
pushed a commit
to Mamoru-Foundation/sui
that referenced
this pull request
Nov 8, 2024
commit 69b5f4dc80d4f5aa488d1102fba32f32b55bfcf9
Author: Carlos Baez <c@mamoru.ai>
Date: Fri Nov 8 16:25:18 2024 +0100
Squashed commit of the following:
commit 7839b9501066108cb2322ba9039120a41781a1b0
Author: John Martin <johnjosephmartin@icloud.com>
Date: Mon Nov 4 14:39:56 2024 -0800
add client-request-method to CORS allowed headers (#20160) (#20168)
## Description
adds `client-request-method` to allowed cors methods
## Test plan
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
## Description
Describe the changes or additions included in this PR.
## Test plan
How did you test the new or updated feature?
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit ecffa454e86b994b79dfc6fb9032d02ee95a0697
Author: John Martin <johnjosephmartin@icloud.com>
Date: Mon Nov 4 13:13:22 2024 -0800
fix: remove high cardinality address label from safe_client_latency (… (#20169)
…#20165)
## Description
safe_client_latency is the highest cardinality metric across our
observability system due to the `address` label + being a histogram:
```
root@sui-node-mysten-rpc-0:/sui# curl -s localhost:9184/metrics | grep safe_client_latency_bucket | wc -l
8512
```
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
## Description
Describe the changes or additions included in this PR.
## Test plan
How did you test the new or updated feature?
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 1493483238672e1d5c22dbff15eb670b1d57883a
Author: Eugene Boguslavsky <eugene@mystenlabs.com>
Date: Mon Nov 4 20:00:02 2024 +0000
Sui v1.37.1 Version Bump
commit 6646b75dd68808b7bedb679c41706b12ae79a5a9
Author: Eugene Boguslavsky <eugene@mystenlabs.com>
Date: Thu Oct 31 18:46:36 2024 -0700
Sui `v1.37.0` Framework Bytecode Snapshot (#20131)
## Description
Sui `v1.37.0` Framework Bytecode Snapshot
## Test plan
`cargo run --bin sui-framework-snapshot`
commit fa5d48fcb3e06a11677dc14e90f12cac59e8f2fc
Author: Ashok Menon <ashok@mystenlabs.com>
Date: Fri Nov 1 01:03:44 2024 +0000
chore(indexer-alt): alphabetize Cargo.toml
commit e116dfa6547e96608bee6767a483e080bb0a1e99
Author: Ashok Menon <ashok@mystenlabs.com>
Date: Fri Nov 1 00:39:20 2024 +0000
indexer-alt: watermark timestamps and other logging improvements (#20120)
## Description
Track the timestamp associated with a watermark, include it in tracing
messages, and introduce gauges to track the timestamps of the latest
checkpoint gathered and written to the DB.
This change also introduces a couple of other timestamp/watermark
related tracing improvements:
- Measure watermarks gathered and in DB for the sequential pipeline
(previously just tracked the watermark after they were written to the
DB). These will usually be the exact same, but may differ if the
pipeline needs to retry a write.
- Gets rid of the ordering and equality impls for `CommitterWatermark`
-- these are no longer required.
- Standardise the ordering of watermark traces between the sequential
and concurrent pipelines, so that they are easier to compare with each
other (they now share a common prefix).
- Standardised tracing messages during committer/watermark teardown: Now
in each exit edge, we log the reason for the exit, and then a message
with the final watermark.
## Test plan
Run `sum_obj_types` and `wal_obj_types` with a consistent range of
`3600` and note that the difference in their timestamps is roughly an
hour (to begin with, it's about 70 minutes):
```
sui$ cargo run -p sui-indexer-alt --release -- \
--database-url 'postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt' \
indexer --remote-store-url 'https://checkpoints.mainnet.sui.io' \
--consistent-range 3600 --pipeline wal_obj_types --pipeline sum_obj_types`
Running `/Users/ashokmenon/sui/idx-poc/target/release/sui-indexer-alt --database-url 'postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt' indexer --remote-store-url 'https://checkpoints.mainnet.sui.io' --consistent-range 3600 --pipeline wal_obj_types --pipeline sum_obj_types`
2024-10-31T15:16:26.867221Z INFO sui_indexer_alt::db: Running migrations ...
2024-10-31T15:16:26.889145Z INFO sui_indexer_alt::db: Migrations complete.
2024-10-31T15:16:27.024406Z INFO sui_indexer_alt: Skipping pipeline ev_emit_mod
2024-10-31T15:16:27.024413Z INFO sui_indexer_alt: Skipping pipeline ev_struct_inst
2024-10-31T15:16:27.024414Z INFO sui_indexer_alt: Skipping pipeline kv_checkpoints
2024-10-31T15:16:27.024414Z INFO sui_indexer_alt: Skipping pipeline kv_objects
2024-10-31T15:16:27.024415Z INFO sui_indexer_alt: Skipping pipeline kv_transactions
2024-10-31T15:16:27.024416Z INFO sui_indexer_alt: Skipping pipeline obj_versions
2024-10-31T15:16:27.024416Z INFO sui_indexer_alt: Skipping pipeline tx_affected_objects
2024-10-31T15:16:27.024417Z INFO sui_indexer_alt: Skipping pipeline tx_balance_changes
2024-10-31T15:16:27.024417Z INFO sui_indexer_alt: Skipping pipeline wal_coin_balances
2024-10-31T15:16:27.033550Z INFO sui_indexer_alt: Skipping pipeline sum_coin_balances
2024-10-31T15:16:27.033607Z INFO sui_indexer_alt::pipeline::processor: Starting processor pipeline="wal_obj_types"
2024-10-31T15:16:27.033856Z INFO sui_indexer_alt::pipeline::concurrent::collector: Starting collector pipeline="wal_obj_types"
2024-10-31T15:16:27.033869Z INFO sui_indexer_alt::pipeline::concurrent::committer: Starting committer pipeline="wal_obj_types"
2024-10-31T15:16:27.033983Z INFO sui_indexer_alt: Skipping pipeline sum_packages
2024-10-31T15:16:27.034011Z INFO sui_indexer_alt::pipeline::processor: Starting processor pipeline="sum_obj_types"
2024-10-31T15:16:27.034183Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Starting watermark pipeline="wal_obj_types" watermark=CommitterWatermark { pipeline: "wal_obj_types", epoch_hi_inclusive: 0, checkpoint_hi_inclusive: 0, tx_hi: 0, timestamp_ms_hi_inclusive: 0 }
2024-10-31T15:16:27.034260Z INFO sui_indexer_alt::pipeline::sequential::committer: Starting committer pipeline="sum_obj_types" watermark=CommitterWatermark { pipeline: "sum_obj_types", epoch_hi_inclusive: 0, checkpoint_hi_inclusive: 0, tx_hi: 0, timestamp_ms_hi_inclusive: 0 }
2024-10-31T15:16:27.034305Z INFO sui_indexer_alt: Ingestion range first_checkpoint=0 last_checkpoint=None
2024-10-31T15:16:27.034339Z INFO sui_indexer_alt::metrics: Starting metrics service on 0.0.0.0:9184
2024-10-31T15:16:27.034351Z INFO sui_indexer_alt::ingestion::regulator: Starting ingestion regulator
2024-10-31T15:16:27.034355Z INFO sui_indexer_alt::ingestion::broadcaster: Starting ingestion broadcaster
2024-10-31T15:16:27.541400Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=212 transaction=213 timestamp=2023-04-13 13:28:25.836 UTC updated=true elapsed_ms=5.486208
2024-10-31T15:16:28.043533Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=515 transaction=516 timestamp=2023-04-13 13:36:17.240 UTC updated=true elapsed_ms=5.512666
2024-10-31T15:16:28.538421Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=1017 transaction=1018 timestamp=2023-04-13 13:48:04.584 UTC updated=true elapsed_ms=3.468209
2024-10-31T15:16:29.041788Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=1422 transaction=1423 timestamp=2023-04-13 13:57:11.577 UTC updated=true elapsed_ms=7.184292
2024-10-31T15:16:29.540675Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=1824 transaction=1825 timestamp=2023-04-13 14:05:59.154 UTC updated=true elapsed_ms=5.7144580000000005
2024-10-31T15:16:30.040987Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=2226 transaction=2227 timestamp=2023-04-13 14:14:54.721 UTC updated=true elapsed_ms=5.955416
2024-10-31T15:16:30.539394Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=2630 transaction=2631 timestamp=2023-04-13 14:23:21.636 UTC updated=true elapsed_ms=5.0003340000000005
2024-10-31T15:16:31.035944Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=3142 transaction=3143 timestamp=2023-04-13 14:34:00.192 UTC updated=true elapsed_ms=1.368042
2024-10-31T15:16:31.532346Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=71 transaction=72 timestamp=2023-04-13 13:24:36.662 UTC
2024-10-31T15:16:31.535237Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=3646 transaction=3647 timestamp=2023-04-13 14:45:19.227 UTC updated=true elapsed_ms=0.7080409999999999
2024-10-31T15:16:31.556459Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=103 transaction=104 timestamp=2023-04-13 13:25:35.491 UTC
2024-10-31T15:16:31.605247Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=151 transaction=152 timestamp=2023-04-13 13:27:01.087 UTC
2024-10-31T15:16:31.658676Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=203 transaction=204 timestamp=2023-04-13 13:28:14.431 UTC
2024-10-31T15:16:31.709093Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=251 transaction=252 timestamp=2023-04-13 13:29:41.350 UTC
2024-10-31T15:16:31.774101Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=304 transaction=305 timestamp=2023-04-13 13:31:05.962 UTC
2024-10-31T15:16:31.823847Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=351 transaction=352 timestamp=2023-04-13 13:32:06.748 UTC
2024-10-31T15:16:31.875613Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=404 transaction=405 timestamp=2023-04-13 13:33:26.122 UTC
2024-10-31T15:16:31.923673Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=452 transaction=453 timestamp=2023-04-13 13:34:41.672 UTC
2024-10-31T15:16:31.973523Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=502 transaction=503 timestamp=2023-04-13 13:35:54.245 UTC
2024-10-31T15:16:32.020968Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=553 transaction=554 timestamp=2023-04-13 13:37:12.723 UTC
2024-10-31T15:16:32.034336Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=4147 transaction=4148 timestamp=2023-04-13 14:56:30.470 UTC updated=true elapsed_ms=0.478916
2024-10-31T15:16:32.081115Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=603 transaction=604 timestamp=2023-04-13 13:38:26.226 UTC
2024-10-31T15:16:32.143228Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=657 transaction=658 timestamp=2023-04-13 13:39:32.583 UTC
2024-10-31T15:16:32.201323Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=710 transaction=711 timestamp=2023-04-13 13:40:44.461 UTC
2024-10-31T15:16:32.239715Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=751 transaction=752 timestamp=2023-04-13 13:41:40.955 UTC
2024-10-31T15:16:32.306828Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=803 transaction=804 timestamp=2023-04-13 13:43:00.433 UTC
2024-10-31T15:16:32.351995Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=854 transaction=855 timestamp=2023-04-13 13:44:21.487 UTC
2024-10-31T15:16:32.408044Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=903 transaction=904 timestamp=2023-04-13 13:45:33.468 UTC
2024-10-31T15:16:32.478623Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=963 transaction=964 timestamp=2023-04-13 13:46:48.531 UTC
2024-10-31T15:16:32.517822Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1004 transaction=1005 timestamp=2023-04-13 13:47:43.997 UTC
2024-10-31T15:16:32.540701Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=4550 transaction=4551 timestamp=2023-04-13 15:05:30.690 UTC updated=true elapsed_ms=5.776375
2024-10-31T15:16:32.574600Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1052 transaction=1053 timestamp=2023-04-13 13:48:55.253 UTC
2024-10-31T15:16:32.625726Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1102 transaction=1103 timestamp=2023-04-13 13:49:57.288 UTC
2024-10-31T15:16:32.679135Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1159 transaction=1160 timestamp=2023-04-13 13:51:06.819 UTC
2024-10-31T15:16:32.729858Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1203 transaction=1204 timestamp=2023-04-13 13:52:11.201 UTC
2024-10-31T15:16:32.777044Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1257 transaction=1258 timestamp=2023-04-13 13:53:26.750 UTC
2024-10-31T15:16:32.819603Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1301 transaction=1302 timestamp=2023-04-13 13:54:17.123 UTC
2024-10-31T15:16:32.870028Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1351 transaction=1352 timestamp=2023-04-13 13:55:24.124 UTC
2024-10-31T15:16:32.923059Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1402 transaction=1403 timestamp=2023-04-13 13:56:49.078 UTC
2024-10-31T15:16:32.969159Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1451 transaction=1452 timestamp=2023-04-13 13:57:56.351 UTC
2024-10-31T15:16:33.034831Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=5055 transaction=5056 timestamp=2023-04-13 15:16:38.038 UTC updated=true elapsed_ms=1.0442500000000001
2024-10-31T15:16:33.036266Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1501 transaction=1502 timestamp=2023-04-13 13:58:56.227 UTC
^C2024-10-31T15:16:33.091189Z INFO sui_indexer_alt::pipeline::concurrent::collector: Shutdown received, stopping collector pipeline="wal_obj_types"
2024-10-31T15:16:33.091195Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Shutdown received pipeline="wal_obj_types"
2024-10-31T15:16:33.091201Z INFO sui_indexer_alt::ingestion::regulator: Shutdown received, stopping regulator
2024-10-31T15:16:33.091213Z INFO sui_indexer_alt::metrics: Shutdown received, stopping metrics service
2024-10-31T15:16:33.091225Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Stopping watermark task pipeline="wal_obj_types" watermark=CommitterWatermark { pipeline: "wal_obj_types", epoch_hi_inclusive: 0, checkpoint_hi_inclusive: 5055, tx_hi: 5056, timestamp_ms_hi_inclusive: 1681398998038 }
2024-10-31T15:16:33.091410Z INFO sui_indexer_alt::pipeline::concurrent::committer: Batches done, stopping committer pipeline="wal_obj_types"
2024-10-31T15:16:33.091721Z INFO sui_indexer_alt::pipeline::sequential::committer: Shutdown received pipeline="sum_obj_types"
2024-10-31T15:16:33.091725Z INFO sui_indexer_alt::pipeline::sequential::committer: Stopping committer pipeline="sum_obj_types" watermark=CommitterWatermark { pipeline: "sum_obj_types", epoch_hi_inclusive: 0, checkpoint_hi_inclusive: 1546, tx_hi: 1547, timestamp_ms_hi_inclusive: 1681394391225 }
2024-10-31T15:16:33.093829Z INFO sui_indexer_alt::pipeline::processor: Shutdown received, stopping processor pipeline="wal_obj_types"
2024-10-31T15:16:33.094014Z INFO sui_indexer_alt::pipeline::processor: Shutdown received, stopping processor pipeline="sum_obj_types"
2024-10-31T15:16:33.094406Z INFO sui_indexer_alt::ingestion::broadcaster: Shutdown received, stopping ingestion broadcaster
2024-10-31T15:16:33.094513Z INFO sui_indexer_alt: Indexing pipeline gracefully shut down
```
## Stack
- #20089
- #20114
- #20116
- #20117
- #20119
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 4b727cbbfd89db7ebfd09b21e05faa58ac2d8c96
Author: Ashok Menon <ashok@mystenlabs.com>
Date: Fri Nov 1 00:37:19 2024 +0000
indexer-alt: allow sequential pipeline immediate reset (#20119)
## Description
If the sequential pipeline committer can guarantee that it could process
more checkpoints by looking at its pending buffer, we now reset the
polling interval immediately, so it does not wait to issue the next
write. This mimics a similar behaviour in the concurrent pipeline.
I made this change after noticing how the pipeline behaves when
ingestion is stuck retrying a checkpoint, on my local machine. Usually
when running locally, performance is limited by checkpoint download
rate, but in a sequential pipeline, if a checkpoint failed to download,
it is possible for many checkpoints to end up processed and pending.
With the previous implementation, once ingestion had recovered (the
checkpoint is fetched), the pending buffer kept growing because it was
only able to land `MAX_BATCH_CHECKPOINTS / commit_interval`, so if
checkpoints were getting added faster than that, it would never recover.
With this change, the pipeline recovers almost instantly, and I expect
that in GCP where bandwidth is not the rate limiting factor, this should
improve throughput during backfill, and synthetic benchmarks.
## Test plan
Run the indexer with a large ingestion buffer and concurrency, wait for
ingestion to fail to fetch a checkpoint, and then notice how the
situation recovers (instead of getting worse until the pipeline
eventually complains that it has too many pending checkpoints):
```
sui$ cargo run -p sui-indexer-alt --release -- \
--database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \
indexer --remote-store-url https://checkpoints.mainnet.sui.io \
--last-checkpoint 1200000 --pipeline sum_packages \
--checkpoint-buffer-size 50000 --ingest-concurrency 20000
```
## Stack
- #20089
- #20114
- #20116
- #20117
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit b65f34c6a00801620e11469575523895a8e074ac
Author: Ashok Menon <ashok@mystenlabs.com>
Date: Fri Nov 1 00:21:38 2024 +0000
indexer-alt: wal_coin_balances pipeline (#20117)
## Description
`wal_coin_balances` is to `sum_coin_balances` what `wal_obj_types` is to
`sum_obj_types`.
## Test plan
Run the indexer, and correlate the live object set calculated from the
write-ahead log against the one that's already in the summary table:
```
sui$ cargo run -p sui-indexer-alt --release -- \
--database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \
indexer --remote-store-url https://checkpoints.mainnet.sui.io/ \
--last-checkpoint 5000
```
```
sui_indexer_alt=# SELECT COUNT(*) FROM sum_coin_balances;
count
-------
178
(1 row)
sui_indexer_alt=# SELECT
COUNT(*)
FROM (
SELECT DISTINCT ON (object_id)
*
FROM
wal_coin_balances
ORDER BY
object_id,
object_version DESC
) o
WHERE
o.owner_id IS NOT NULL;
count
-------
178
(1 row)
```
## Stack
- #20089
- #20114
- #20116
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 71f1faa1a8557a178e0bf89f20829a8ea503f156
Author: Ashok Menon <ashok@mystenlabs.com>
Date: Fri Nov 1 00:18:46 2024 +0000
indexer-alt: wal_obj_types pipeline (#20116)
## Description
Adds the concurrent pipeline that writes the write-ahead log for
`sum_obj_types`. It re-uses the `process` implementation from
`sum_obj_types` and then writes it into an append-only table.
Note that today:
- The pipelines are completely independent, which means the WAL pipeline
redoes the processing work of the summary pipeline (this is presumably
not an issue because the repeated work is not particularly heavy).
- This change does not include the pruning necessary to keep this
table's size in check (in practice it should only be a couple of gigs in
size). This will come in a follow-up PR.
## Test plan
Run the indexer and cross check the live-object set calculated from the
write-ahead log with the summary:
```
sui$ cargo run -p sui-indexer-alt --release -- \
--database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \
indexer --remote-store-url https://checkpoints.mainnet.sui.io \
--last-checkpoint 5000
```
```
sui_indexer_alt=# SELECT COUNT(*) FROM sum_obj_types;
count
-------
592
(1 row)
^
sui_indexer_alt=# SELECT
COUNT(*)
FROM (
SELECT DISTINCT ON (object_id)
*
FROM
wal_obj_types
ORDER BY
object_id,
object_version DESC
) o
WHERE
o.owner_kind IS NOT NULL;
count
-------
592
(1 row)
```
## Stack
- #20089
- #20114
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 7de02417c27844c8a086866b1279b4736ab862ec
Author: Ashok Menon <ashok@mystenlabs.com>
Date: Thu Oct 31 23:22:42 2024 +0000
indexer-alt: obj_versions pipeline (#20114)
## Description
Pipeline to fill `obj_versions` table -- this is the table that serves
"parent version" (dynamic field) queries.
## Test plan
```
sui$ cargo run -p sui-indexer-alt --release -- \
--database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \
indexer --remote-url "https://checkpoints.mainnet.sui.io" \
--last-checkpoint 5000
```
## Stack
- #20089
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 53eaed5e300a67f0d01cf68119a145410925f9de
Author: Ashok Menon <ashok@mystenlabs.com>
Date: Wed Oct 30 00:06:07 2024 +0000
indexer-alt: delays for sequential pipelines
## Description
Add the ability to include an optional delay to sequential pipelines.
This can be used to implement behaviour similar to `objects_snapshot` in
the new architecture.
Sequential pipelines honour this configuration in their committer tasks:
- They process checkpoints at the same rate as other pipelines
- But hold back data in the committer's "pending" map until the
committer witnesses a checkpoint that is at least `checkpoint_lag`
checkpoints ahead.
This also requires slightly complicating the set-up and tear-down logic:
- When setting up, we may need to wait until the pending buffer has been
primed enough to set a valid commit upperbound.
- If the pipeline runs dry before that point, we need to add an extra
check to close down the committer.
- When tearing down the committer after its channel has closed, we need
to check whether batching would have picked up the first pending
checkpoint (whether it is a valid "next checkpoint" and it's not held
up because of a lag parameter).
## Test plan
Run the indexer with various parameterisations for the consistent range,
and note that the watermark for the summary pipelines lag behind the
watermarks for other pipelines.
```
sui$ cargo run -p sui-indexer-alt --release -- \
--database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \
indexer --remote-store-url https://checkpoints.mainnet.sui.io \
--pipeline kv_objects \
--pipeline sum_obj_types --pipeline sum_coin_balances \
--last-checkpoint 5000 --consistent-range 1000
[...]
2024-10-30T00:24:45.148259Z INFO sui_indexer_alt::ingestion::regulator: Checkpoints done, stopping regulator
2024-10-30T00:24:45.161682Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=3605 transaction=3606
2024-10-30T00:24:45.212394Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_coin_balances" epoch=0 checkpoint=2703 transaction=2704
2024-10-30T00:24:45.222986Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=3656 transaction=3657
2024-10-30T00:24:45.274318Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=3702 transaction=3703
2024-10-30T00:24:45.340714Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=3753 transaction=3754
2024-10-30T00:24:45.355934Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="kv_objects" elapsed_ms=0.949083 updated=true epoch=0 checkpoint=4760 transaction=4761
2024-10-30T00:24:45.401262Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=3801 transaction=3802
2024-10-30T00:24:45.463134Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=3852 transaction=3853
2024-10-30T00:24:45.544933Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=3904 transaction=3905
2024-10-30T00:24:45.609068Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=3952 transaction=3953
2024-10-30T00:24:45.668560Z INFO sui_indexer_alt::ingestion::broadcaster: Checkpoints done, stopping ingestion broadcaster
2024-10-30T00:24:45.668593Z INFO sui_indexer_alt::pipeline::processor: Checkpoints done, stopping processor pipeline="kv_objects"
2024-10-30T00:24:45.668605Z INFO sui_indexer_alt::pipeline::processor: Checkpoints done, stopping processor pipeline="sum_obj_types"
2024-10-30T00:24:45.668605Z INFO sui_indexer_alt::pipeline::processor: Checkpoints done, stopping processor pipeline="sum_coin_balances"
2024-10-30T00:24:45.674305Z INFO sui_indexer_alt::pipeline::sequential::committer: Processor closed channel, pending rows empty, stopping committer pipeline="sum_obj_types" watermark=CommitterWatermark { pipeline: "sum_obj_types", epoch_hi_inclusive: 0, checkpoint_hi_inclusive: 4000, tx_hi: 4001 }
2024-10-30T00:24:45.713856Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_coin_balances" epoch=0 checkpoint=3003 transaction=3004
2024-10-30T00:24:45.863837Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="kv_objects" elapsed_ms=1.83825 updated=true epoch=0 checkpoint=4960 transaction=4961
2024-10-30T00:24:46.120459Z INFO sui_indexer_alt::pipeline::concurrent::collector: Processor closed channel, pending rows empty, stopping collector pipeline="kv_objects"
2024-10-30T00:24:46.131407Z INFO sui_indexer_alt::pipeline::concurrent::committer: Batches done, stopping committer pipeline="kv_objects"
2024-10-30T00:24:46.214267Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_coin_balances" epoch=0 checkpoint=3303 transaction=3304
2024-10-30T00:24:46.356941Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="kv_objects" elapsed_ms=0.68 updated=true epoch=0 checkpoint=5000 transaction=5001
2024-10-30T00:24:46.356970Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Committer closed channel, stopping watermark task pipeline="kv_objects" watermark=CommitterWatermark { pipeline: "kv_objects", epoch_hi_inclusive: 0, checkpoint_hi_inclusive: 5000, tx_hi: 5001 }
2024-10-30T00:24:46.714498Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_coin_balances" epoch=0 checkpoint=3603 transaction=3604
2024-10-30T00:24:47.214758Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_coin_balances" epoch=0 checkpoint=3903 transaction=3904
2024-10-30T00:24:47.715841Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_coin_balances" epoch=0 checkpoint=4000 transaction=4001
2024-10-30T00:24:47.715917Z INFO sui_indexer_alt::pipeline::sequential::committer: Processor closed channel, pending rows empty, stopping committer pipeline="sum_coin_balances" watermark=CommitterWatermark { pipeline: "sum_coin_balances", epoch_hi_inclusive: 0, checkpoint_hi_inclusive: 4000, tx_hi: 4001 }
2024-10-30T00:24:47.716485Z INFO sui_indexer_alt: Indexing pipeline gracefully shut down
2024-10-30T00:24:47.716647Z INFO sui_indexer_alt::metrics: Shutdown received, stopping metrics service
```
commit d57326492b62ec8ba7ff8826ad473ec1b5008f11
Author: Ashok Menon <ashok@mystenlabs.com>
Date: Wed Oct 30 00:01:45 2024 +0000
fix(indexer-alt): remove double logging on pipeline skip
## Description
We were logging the pipeline being skipped twice.
## Test plan
Check logs while the indexer starts up:
```
sui$ cargo run -p sui-indexer-alt --release -- \
--database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \
indexer --remote-store-url "https://checkpoints.mainnet.sui.io" \
--last-checkpoint 5000 --pipeline sum_obj_types
```
commit ef0d78c638e3ef0d0a3b28787b618ea56a140284
Author: Ge Gao <106119108+gegaowp@users.noreply.github.com>
Date: Thu Oct 31 19:30:51 2024 -0400
add sui-field-count for sui-indexer-alt (#20126)
## Description
title, so that we can replace hard-coded field count in pipelines.
a crate did very similar things but it's archived, so prob better to
have our own impl, also b/c the functionality is relatively simple
https://github.com/discosultan/field-count
## Test plan
added tests in models/
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit c92dd1b4f16074ced8c1fe5c56a10e675f68bbe6
Author: phoenix <51927076+phoenix-o@users.noreply.github.com>
Date: Thu Oct 31 18:28:34 2024 -0400
[kv store] bigtable. Add support for querying checkpoint by digest (#20122)
## Description
Adds support for querying checkpoint by digest
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 74ace7b6166f10bddfd714568e0ddc20db2ca75d
Author: JasonRUAN <rzexin@gmail.com>
Date: Fri Nov 1 06:25:44 2024 +0800
[CLI] fix `sui keytool import` --alias command and default alias does not work (#20111)
## Description
fix `sui keytool import` --alias command and default alias does not work
Before Fix：
<img width="1295" alt="image"
src="https://github.com/user-attachments/assets/4cb40d92-e266-4b6e-9c39-c05ac52d7a9d">
After Fix：
- **--alias**
<img width="1301" alt="image"
src="https://github.com/user-attachments/assets/b5c711e8-2a4d-4861-9965-30cde260e93f">
- **default alias**
<img width="1295" alt="image"
src="https://github.com/user-attachments/assets/4513db65-0e10-4179-b5f0-de188871cf78">
## Test plan
How did you test the new or updated feature?
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [x] CLI: `sui keytool import --alias` is now fixed and correctly
records the provided alias.
- [ ] Rust SDK:
- [ ] REST API:
commit e41a759769fdea08f660160a23e945bbad80b337
Author: sui-merge-bot[bot] <114704316+sui-merge-bot[bot]@users.noreply.github.com>
Date: Thu Oct 31 23:50:37 2024 +0200
Version Packages (#20127)
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.
# Releases
## @mysten/kms@0.0.3
### Patch Changes
- 02c9e46: Fix exports on the bundled package
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
commit 02c9e4691b429b4a0ec85b71eace37b72515d055
Author: Manolis Liolios <manos@mystenlabs.com>
Date: Thu Oct 31 23:38:15 2024 +0200
Fix build, re-publish @mysten/kms (#20125)
## Description
Build will always treat `src` as the base for outputs when building.
## Test plan
How did you test the new or updated feature?
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 16f2683a99e7b8824aa5879d3f20ffef936a7546
Author: William Smith <williamprincesmith@gmail.com>
Date: Thu Oct 31 17:27:16 2024 -0400
[sui-benchmark] Add expected failure payloads (#19831)
## Description
Introduce a new payload type into the benchmark tool -
`ExpectedFailurePayload`.
This payload type is configured with some expected failure type
(currently only one implemented - user signature failure - but others
can be added with minimal scaffolding) such that
`payload.make_transaction()` generates a transaction that will fail in
this manner. Note that the failures that this payload type is concerned
with are failures to execute the transaction itself, rather than during
execution by the MoveVM. In other words, it is expected that the failure
mode will not consume gas or produce effects.
Note that for this reason, its failure mode is inverted. It will be
tallied for metrics purposes as an `error` if it succeeds and tallied as
an `expected_error` (which is a success) if it fails.
Also note that transaction responses for this type are handled by
producing a `NextOp::Retry` (with some additional logging/metrics) since
it is functionally equivalent to a retryable error.
## Test plan
Ran the following:
```
SIM_STRESS_TEST_QPS=200 SIM_STRESS_TEST_WORKERS=20 RUST_LOG=debug cargo simtest --nocapture test_simulated_load_expected_failure_traffic_control
```
Observed the following in logs
```
2022-01-03T02:05:47.734507Z INFO node{id=1 name="client"}: simtest::test: crates/sui-benchmark/tests/simtest.rs:1088: end of test BenchmarkStats { duration: 50.000002242s, num_error_txes: 0, num_expected_error_txes: 8049, num_success_txes: 348, num_success_cmds: 1977, total_gas_used: 3215776400, latency_ms: HistogramWrapper { histogram: Histogram { auto_resize: false, highest_trackable_value: 120000, lowest_discernible_value: 1, significant_value_digits: 3, bucket_count: 7, sub_bucket_count: 2048, sub_bucket_half_count: 1024, sub_bucket_half_count_magnitude: 10, sub_bucket_mask: 2047, leading_zero_count_base: 53, unit_magnitude: 0, unit_magnitude_mask: 0, max_value: 14359, min_non_zero_value: 72, total_count: 348, counts: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 8, 12, 9, 20, 25, 22, 19
...
```
[Also ran in
PTN](https://metrics.sui.io/d/adl51ctsvmkg0a/traffic-control-dos-protection-dashboard?from=2024-10-31T19:00:50.548Z&to=2024-10-31T20:20:03.673Z&timezone=browser&var-network=private-testnet&var-host=$__all&var-fullnode=$__all)
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit d95918cea0d8fa0f1a4007273f477e3c12b61302
Author: phoenix <51927076+phoenix-o@users.noreply.github.com>
Date: Thu Oct 31 17:18:17 2024 -0400
[data ingestion] add support for bigtable in main workflow binary (#20018)
## Description
adding BigTable KV as a variant to main binary that manages internal
workflows
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit c33374e735257b2a460a2f882e062619f8958bc5
Author: wlmyng <127570466+wlmyng@users.noreply.github.com>
Date: Thu Oct 31 12:51:47 2024 -0700
[mvr] indexer schema subset and reduced persist tasks (#20100)
## Description
Describe the changes or additions included in this PR.
## Test plan
How did you test the new or updated feature?
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 7c517e7e5ce7193d1f51ef7f20a31b56f0e66797
Author: Xun Li <lxfind@gmail.com>
Date: Thu Oct 31 12:49:58 2024 -0700
Add system_invariant_violation macro (#20105)
## Description
Introduce a system invariant violation macro.
We can then set up alerts whenever the counter value changes.
Used it in one place as an example.
## Test plan
How did you test the new or updated feature?
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit b248f35879801b3bae9b76c4d204f36540d816f8
Author: Andrew Schran <aschran@mystenlabs.com>
Date: Thu Oct 31 14:16:48 2024 -0400
Add support for TLS connections with self-signed cert on validator gRPC interface (#19796)
Client side use of TLS will be enabled-by-default in a future PR.
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [x] Nodes (Validators and Full nodes): Adds support for TLS on
validator gRPC interface.
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit e06610823b985bdc01c40e23c9a9a2f8d5b571d1
Author: sui-merge-bot[bot] <114704316+sui-merge-bot[bot]@users.noreply.github.com>
Date: Thu Oct 31 19:58:58 2024 +0200
Version Packages (#20076)
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.
# Releases
## @mysten/sui@1.14.0
### Minor Changes
- c24814b: Adds a custom header; 'Client-Request-Method' which will
contain the method name used in
each outgoing jsonrpc request
## @mysten/create-dapp@0.3.28
### Patch Changes
- Updated dependencies [c24814b]
- @mysten/sui@1.14.0
- @mysten/dapp-kit@0.14.28
## @mysten/dapp-kit@0.14.28
### Patch Changes
- Updated dependencies [c24814b]
- @mysten/sui@1.14.0
- @mysten/wallet-standard@0.13.9
- @mysten/zksend@0.11.9
## @mysten/deepbook@0.8.23
### Patch Changes
- Updated dependencies [c24814b]
- @mysten/sui@1.14.0
## @mysten/deepbook-v3@0.12.2
### Patch Changes
- Updated dependencies [c24814b]
- @mysten/sui@1.14.0
## @mysten/enoki@0.4.7
### Patch Changes
- Updated dependencies [c24814b]
- @mysten/sui@1.14.0
- @mysten/zklogin@0.7.24
## @mysten/graphql-transport@0.2.25
### Patch Changes
- Updated dependencies [c24814b]
- @mysten/sui@1.14.0
## @mysten/kiosk@0.9.23
### Patch Changes
- 4166d71: Fix doc comment on `getKiosk` command
- Updated dependencies [c24814b]
- @mysten/sui@1.14.0
## @mysten/kms@0.0.2
### Patch Changes
- b3f3925: Introduces @mysten/kms which initially exposes a Sui AWS KMS
signer
- Updated dependencies [c24814b]
- @mysten/sui@1.14.0
## @mysten/suins-toolkit@0.5.23
### Patch Changes
- Updated dependencies [c24814b]
- @mysten/sui@1.14.0
## @mysten/wallet-standard@0.13.9
### Patch Changes
- Updated dependencies [c24814b]
- @mysten/sui@1.14.0
## @mysten/zklogin@0.7.24
### Patch Changes
- Updated dependencies [c24814b]
- @mysten/sui@1.14.0
## @mysten/zksend@0.11.9
### Patch Changes
- Updated dependencies [c24814b]
- @mysten/sui@1.14.0
- @mysten/wallet-standard@0.13.9
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
commit 093954a09bce1942a9f9773b845f59bf5124efbf
Author: ronny-mysten <118224482+ronny-mysten@users.noreply.github.com>
Date: Thu Oct 31 11:35:27 2024 -0600
[docs] Rpc update (#20094)
## Description
Updates the RPC render to include component schemas. Also updates a few
comments.
## Test plan
How did you test the new or updated feature?
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit fa66617b7dfa7096fd64a2dc1b002d154223be4a
Author: wlmyng <127570466+wlmyng@users.noreply.github.com>
Date: Thu Oct 31 10:18:57 2024 -0700
[mvr] renaming to sui-mvr-indexer and Dockerfile (#20099)
## Description
Describe the changes or additions included in this PR.
## Test plan
How did you test the new or updated feature?
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 434206f14a0a21bc62ea08480b4a5c4d36d6cc25
Author: Todd Nowacki <tmn@mystenlabs.com>
Date: Thu Oct 31 10:18:46 2024 -0700
[move-compiler] Re-organize warning filters (#20103)
## Description
- Re-organize warning filters. It should make it a bit cleaner to
refactor diagnostic reporting
## Test plan
- ran tests
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 0d16b6f8b7c4ff79f70400e8a94d7486b2b2c3fe
Author: wlmyng <127570466+wlmyng@users.noreply.github.com>
Date: Thu Oct 31 09:36:37 2024 -0700
[mvr] new crate for custom mvr-indexer, copied from sui-indexer crate (#20098)
## Description
Simple copy-paste of the sui-indexer crate for the custom mvr indexer
## Test plan
How did you test the new or updated feature?
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit b3f39253ff0153b69b42bc298d1ecdbb0524e276
Author: John Naulty Jr. <jnaulty@mystenlabs.com>
Date: Thu Oct 31 09:21:29 2024 -0700
AWS KMS TS Signer (#16139)
## Description
Describe the changes or additions included in this PR.
## Test Plan
How did you test the new or updated feature?
---
If your changes are not user-facing and do not break anything, you can
skip the following section. Otherwise, please briefly describe what has
changed under the Release Notes section.
### Type of Change (Check all that apply)
- [ ] protocol change
- [ ] user-visible impact
- [ ] breaking change for a client SDKs
- [ ] breaking change for FNs (FN binary must upgrade)
- [ ] breaking change for validators or node operators (must upgrade
binaries)
- [ ] breaking change for on-chain data layout
- [ ] necessitate either a data wipe or data migration
### Release notes
---------
Co-authored-by: Manolis Liolios <manos@mystenlabs.com>
commit f78c7b006c28c4c1afd3f7d42e21e53970902f5d
Author: Brandon Williams <brandon@mystenlabs.com>
Date: Wed Oct 30 19:50:45 2024 -0500
rest: revert client change to requested protobuf and request bcs instead
Revert the client change to request protobuf as a response type, and
instead request bcs, because the support for protobuf hasn't rolled out
to other networks yet.
commit 108ad4c8ea53c68bd90d0b3fcdc88fbc75640c61
Author: Brandon Williams <brandon@mystenlabs.com>
Date: Wed Oct 30 12:50:43 2024 -0500
rest: support protobuf format for committee endpoints
commit 718a244bca584382faedde56f8247c6b26225b07
Author: Brandon Williams <brandon@mystenlabs.com>
Date: Wed Oct 30 11:03:43 2024 -0500
rest: support protobuf format for resolve, simulate, and execute transaction endpoints
commit 2987f5967ecffa8ddbec2b8ebf9ee2eec09dad60
Author: Brandon Williams <brandon@mystenlabs.com>
Date: Wed Oct 30 08:55:51 2024 -0500
rest: support protobuf format for GetTransaction and ListTransactions endpoints
commit 672ecbd2a58c7514aa24d139c442316a03d7f89e
Author: Brandon Williams <brandon@mystenlabs.com>
Date: Tue Oct 29 21:30:42 2024 -0500
rest: support protobuf format for GetObject endpoints
commit c8472c057523ff88040ae688f1410d34164b0ebc
Author: Brandon Williams <brandon@mystenlabs.com>
Date: Tue Oct 29 19:00:39 2024 -0500
rest: remove ListFullCheckpoint api
Remove the ListFullCheckpoint api as its a bit too heavy-weight and
instead we'll look to introduce a streaming api in the future.
commit 39f1ae21fd5467e71522729519720d2add9468dc
Author: Xun Li <lxfind@gmail.com>
Date: Wed Oct 30 22:25:50 2024 -0700
Clean up local execution (#20086)
## Description
Describe the changes or additions included in this PR.
## Test plan
How did you test the new or updated feature?
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 65b02e203ad897f438f87d0e5cfe46c56a1d4247
Author: phoenix <51927076+phoenix-o@users.noreply.github.com>
Date: Wed Oct 30 21:26:54 2024 -0400
[pruner] enable periodic compaction of individual SST files by default (#20101)
## Description
enables periodic compaction of individual SST files by default
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit f2670718f694edd59e8a062bc932d4efd96ba765
Author: Tim Zakian <2895723+tzakian@users.noreply.github.com>
Date: Wed Oct 30 16:49:01 2024 -0700
[execution] Update to deps-only mode (#20102)
## Description
Disallow new modules from being added to packages in deps-only mode.
## Test plan
Added new tests to check for both previous and new behavior.
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [X] Protocol: Updated restrictions on deps-only packages to not allow
adding new modules to the package.
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 7e6319fa33b7021a94ff7dcc07e479a4bae92b55
Author: Adam Welc <adam@mystenlabs.com>
Date: Wed Oct 30 15:51:37 2024 -0700
[trace-view] Added support for native functions and global locations (#20080)
## Description
Adds support for native functions where the `CloseFrame` event happens
right after `OpenFrame` event and both need to be skipped to skip over
the native function. Also added support for handling global locations -
they should be parsed properly but largely ignored in the trace as they
cannot directly affect values of local variables
## Test plan
All old and new tests must pass
commit 21639a51847f1a04b6b221ac77df3875c0430be7
Author: Xun Li <lxfind@gmail.com>
Date: Wed Oct 30 14:30:59 2024 -0700
[RPC] Fix coin metadata api inefficiency (#19794)
## Description
This PR fixes a minor inefficiency in the coin metadata api.
It does not need to fetch the transaction, as it only needs the effects.
This likely avoids a remote KV fetch.
## Test plan
CI
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after t…
carlosbmamoru
pushed a commit
to Mamoru-Foundation/sui
that referenced
this pull request
Nov 14, 2024
commit dc117c586a01d6dbef41eafcaf3d0be2e37f2d12
Merge: 32514d1c36 b8eb8920ae
Author: Carlos Baez <c@mevton.com>
Date: Thu Nov 14 11:36:36 2024 +0100
Merge tag 'testnet-v1.37.3' into testnet4-wit-v1.37.3
Sui testnet v1.37.3 release
commit b8eb8920aeca592d0cf15fbddc7f64ed4aad7202
Author: Andrew Schran <aschran@mystenlabs.com>
Date: Tue Nov 12 15:34:20 2024 -0500
Revert "Add support for TLS connections with self-signed cert on validator gRPC interface" (#20232) (#20234)
Reverts MystenLabs/sui#19796 due to suspicion it's causing performance
issues
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit e205b302bd5c5742d21fbc18811b705980714c81
Author: Eugene Boguslavsky <eugene@mystenlabs.com>
Date: Tue Nov 12 16:56:05 2024 +0000
Sui v1.37.3 Version Bump
commit f77489d1cf177b6bacda47c3c3ec6f120d1fed45
Author: Emma Zhong <emma@mystenlabs.com>
Date: Mon Nov 11 19:22:47 2024 -0800
[cherrypick][jsonrpc] fix estimated rewards during safe mode (#20182) (#20187)
## Description
Currently if the exchange rate for an epoch is not found, we would
assign it the default exchange rate while we should've used the exchange
rate from the previous existing epoch instead. This PR fixes that.
## Test plan
Added some unit tests.
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [x] JSON-RPC: fixed reward estimation in `getStakes` API so that we
don't overestimate stake rewards of stakes activated during safe mode
epochs.
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
## Description
Describe the changes or additions included in this PR.
## Test plan
How did you test the new or updated feature?
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 102280ae7591ec1cf187a433624443b8798e5605
Author: Eugene Boguslavsky <eugene@mystenlabs.com>
Date: Thu Nov 7 19:25:00 2024 +0000
Sui v1.37.2 Version Bump
commit b15a076619641002381f4be001f28be5dce7a3e9
Author: wlmyng <127570466+wlmyng@users.noreply.github.com>
Date: Tue Nov 5 14:18:50 2024 -0800
[graphql][cherry-pick] need to add object_status=0 to use partial indexes (#20176) (#20181)
## Description
cherry pick
## Test plan
How did you test the new or updated feature?
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 7839b9501066108cb2322ba9039120a41781a1b0
Author: John Martin <johnjosephmartin@icloud.com>
Date: Mon Nov 4 14:39:56 2024 -0800
add client-request-method to CORS allowed headers (#20160) (#20168)
## Description
adds `client-request-method` to allowed cors methods
## Test plan
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
## Description
Describe the changes or additions included in this PR.
## Test plan
How did you test the new or updated feature?
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit ecffa454e86b994b79dfc6fb9032d02ee95a0697
Author: John Martin <johnjosephmartin@icloud.com>
Date: Mon Nov 4 13:13:22 2024 -0800
fix: remove high cardinality address label from safe_client_latency (… (#20169)
…#20165)
## Description
safe_client_latency is the highest cardinality metric across our
observability system due to the `address` label + being a histogram:
```
root@sui-node-mysten-rpc-0:/sui# curl -s localhost:9184/metrics | grep safe_client_latency_bucket | wc -l
8512
```
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
## Description
Describe the changes or additions included in this PR.
## Test plan
How did you test the new or updated feature?
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 1493483238672e1d5c22dbff15eb670b1d57883a
Author: Eugene Boguslavsky <eugene@mystenlabs.com>
Date: Mon Nov 4 20:00:02 2024 +0000
Sui v1.37.1 Version Bump
commit 6646b75dd68808b7bedb679c41706b12ae79a5a9
Author: Eugene Boguslavsky <eugene@mystenlabs.com>
Date: Thu Oct 31 18:46:36 2024 -0700
Sui `v1.37.0` Framework Bytecode Snapshot (#20131)
## Description
Sui `v1.37.0` Framework Bytecode Snapshot
## Test plan
`cargo run --bin sui-framework-snapshot`
commit fa5d48fcb3e06a11677dc14e90f12cac59e8f2fc
Author: Ashok Menon <ashok@mystenlabs.com>
Date: Fri Nov 1 01:03:44 2024 +0000
chore(indexer-alt): alphabetize Cargo.toml
commit e116dfa6547e96608bee6767a483e080bb0a1e99
Author: Ashok Menon <ashok@mystenlabs.com>
Date: Fri Nov 1 00:39:20 2024 +0000
indexer-alt: watermark timestamps and other logging improvements (#20120)
## Description
Track the timestamp associated with a watermark, include it in tracing
messages, and introduce gauges to track the timestamps of the latest
checkpoint gathered and written to the DB.
This change also introduces a couple of other timestamp/watermark
related tracing improvements:
- Measure watermarks gathered and in DB for the sequential pipeline
(previously just tracked the watermark after they were written to the
DB). These will usually be the exact same, but may differ if the
pipeline needs to retry a write.
- Gets rid of the ordering and equality impls for `CommitterWatermark`
-- these are no longer required.
- Standardise the ordering of watermark traces between the sequential
and concurrent pipelines, so that they are easier to compare with each
other (they now share a common prefix).
- Standardised tracing messages during committer/watermark teardown: Now
in each exit edge, we log the reason for the exit, and then a message
with the final watermark.
## Test plan
Run `sum_obj_types` and `wal_obj_types` with a consistent range of
`3600` and note that the difference in their timestamps is roughly an
hour (to begin with, it's about 70 minutes):
```
sui$ cargo run -p sui-indexer-alt --release -- \
--database-url 'postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt' \
indexer --remote-store-url 'https://checkpoints.mainnet.sui.io' \
--consistent-range 3600 --pipeline wal_obj_types --pipeline sum_obj_types`
Running `/Users/ashokmenon/sui/idx-poc/target/release/sui-indexer-alt --database-url 'postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt' indexer --remote-store-url 'https://checkpoints.mainnet.sui.io' --consistent-range 3600 --pipeline wal_obj_types --pipeline sum_obj_types`
2024-10-31T15:16:26.867221Z INFO sui_indexer_alt::db: Running migrations ...
2024-10-31T15:16:26.889145Z INFO sui_indexer_alt::db: Migrations complete.
2024-10-31T15:16:27.024406Z INFO sui_indexer_alt: Skipping pipeline ev_emit_mod
2024-10-31T15:16:27.024413Z INFO sui_indexer_alt: Skipping pipeline ev_struct_inst
2024-10-31T15:16:27.024414Z INFO sui_indexer_alt: Skipping pipeline kv_checkpoints
2024-10-31T15:16:27.024414Z INFO sui_indexer_alt: Skipping pipeline kv_objects
2024-10-31T15:16:27.024415Z INFO sui_indexer_alt: Skipping pipeline kv_transactions
2024-10-31T15:16:27.024416Z INFO sui_indexer_alt: Skipping pipeline obj_versions
2024-10-31T15:16:27.024416Z INFO sui_indexer_alt: Skipping pipeline tx_affected_objects
2024-10-31T15:16:27.024417Z INFO sui_indexer_alt: Skipping pipeline tx_balance_changes
2024-10-31T15:16:27.024417Z INFO sui_indexer_alt: Skipping pipeline wal_coin_balances
2024-10-31T15:16:27.033550Z INFO sui_indexer_alt: Skipping pipeline sum_coin_balances
2024-10-31T15:16:27.033607Z INFO sui_indexer_alt::pipeline::processor: Starting processor pipeline="wal_obj_types"
2024-10-31T15:16:27.033856Z INFO sui_indexer_alt::pipeline::concurrent::collector: Starting collector pipeline="wal_obj_types"
2024-10-31T15:16:27.033869Z INFO sui_indexer_alt::pipeline::concurrent::committer: Starting committer pipeline="wal_obj_types"
2024-10-31T15:16:27.033983Z INFO sui_indexer_alt: Skipping pipeline sum_packages
2024-10-31T15:16:27.034011Z INFO sui_indexer_alt::pipeline::processor: Starting processor pipeline="sum_obj_types"
2024-10-31T15:16:27.034183Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Starting watermark pipeline="wal_obj_types" watermark=CommitterWatermark { pipeline: "wal_obj_types", epoch_hi_inclusive: 0, checkpoint_hi_inclusive: 0, tx_hi: 0, timestamp_ms_hi_inclusive: 0 }
2024-10-31T15:16:27.034260Z INFO sui_indexer_alt::pipeline::sequential::committer: Starting committer pipeline="sum_obj_types" watermark=CommitterWatermark { pipeline: "sum_obj_types", epoch_hi_inclusive: 0, checkpoint_hi_inclusive: 0, tx_hi: 0, timestamp_ms_hi_inclusive: 0 }
2024-10-31T15:16:27.034305Z INFO sui_indexer_alt: Ingestion range first_checkpoint=0 last_checkpoint=None
2024-10-31T15:16:27.034339Z INFO sui_indexer_alt::metrics: Starting metrics service on 0.0.0.0:9184
2024-10-31T15:16:27.034351Z INFO sui_indexer_alt::ingestion::regulator: Starting ingestion regulator
2024-10-31T15:16:27.034355Z INFO sui_indexer_alt::ingestion::broadcaster: Starting ingestion broadcaster
2024-10-31T15:16:27.541400Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=212 transaction=213 timestamp=2023-04-13 13:28:25.836 UTC updated=true elapsed_ms=5.486208
2024-10-31T15:16:28.043533Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=515 transaction=516 timestamp=2023-04-13 13:36:17.240 UTC updated=true elapsed_ms=5.512666
2024-10-31T15:16:28.538421Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=1017 transaction=1018 timestamp=2023-04-13 13:48:04.584 UTC updated=true elapsed_ms=3.468209
2024-10-31T15:16:29.041788Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=1422 transaction=1423 timestamp=2023-04-13 13:57:11.577 UTC updated=true elapsed_ms=7.184292
2024-10-31T15:16:29.540675Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=1824 transaction=1825 timestamp=2023-04-13 14:05:59.154 UTC updated=true elapsed_ms=5.7144580000000005
2024-10-31T15:16:30.040987Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=2226 transaction=2227 timestamp=2023-04-13 14:14:54.721 UTC updated=true elapsed_ms=5.955416
2024-10-31T15:16:30.539394Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=2630 transaction=2631 timestamp=2023-04-13 14:23:21.636 UTC updated=true elapsed_ms=5.0003340000000005
2024-10-31T15:16:31.035944Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=3142 transaction=3143 timestamp=2023-04-13 14:34:00.192 UTC updated=true elapsed_ms=1.368042
2024-10-31T15:16:31.532346Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=71 transaction=72 timestamp=2023-04-13 13:24:36.662 UTC
2024-10-31T15:16:31.535237Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=3646 transaction=3647 timestamp=2023-04-13 14:45:19.227 UTC updated=true elapsed_ms=0.7080409999999999
2024-10-31T15:16:31.556459Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=103 transaction=104 timestamp=2023-04-13 13:25:35.491 UTC
2024-10-31T15:16:31.605247Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=151 transaction=152 timestamp=2023-04-13 13:27:01.087 UTC
2024-10-31T15:16:31.658676Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=203 transaction=204 timestamp=2023-04-13 13:28:14.431 UTC
2024-10-31T15:16:31.709093Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=251 transaction=252 timestamp=2023-04-13 13:29:41.350 UTC
2024-10-31T15:16:31.774101Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=304 transaction=305 timestamp=2023-04-13 13:31:05.962 UTC
2024-10-31T15:16:31.823847Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=351 transaction=352 timestamp=2023-04-13 13:32:06.748 UTC
2024-10-31T15:16:31.875613Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=404 transaction=405 timestamp=2023-04-13 13:33:26.122 UTC
2024-10-31T15:16:31.923673Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=452 transaction=453 timestamp=2023-04-13 13:34:41.672 UTC
2024-10-31T15:16:31.973523Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=502 transaction=503 timestamp=2023-04-13 13:35:54.245 UTC
2024-10-31T15:16:32.020968Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=553 transaction=554 timestamp=2023-04-13 13:37:12.723 UTC
2024-10-31T15:16:32.034336Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=4147 transaction=4148 timestamp=2023-04-13 14:56:30.470 UTC updated=true elapsed_ms=0.478916
2024-10-31T15:16:32.081115Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=603 transaction=604 timestamp=2023-04-13 13:38:26.226 UTC
2024-10-31T15:16:32.143228Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=657 transaction=658 timestamp=2023-04-13 13:39:32.583 UTC
2024-10-31T15:16:32.201323Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=710 transaction=711 timestamp=2023-04-13 13:40:44.461 UTC
2024-10-31T15:16:32.239715Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=751 transaction=752 timestamp=2023-04-13 13:41:40.955 UTC
2024-10-31T15:16:32.306828Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=803 transaction=804 timestamp=2023-04-13 13:43:00.433 UTC
2024-10-31T15:16:32.351995Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=854 transaction=855 timestamp=2023-04-13 13:44:21.487 UTC
2024-10-31T15:16:32.408044Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=903 transaction=904 timestamp=2023-04-13 13:45:33.468 UTC
2024-10-31T15:16:32.478623Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=963 transaction=964 timestamp=2023-04-13 13:46:48.531 UTC
2024-10-31T15:16:32.517822Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1004 transaction=1005 timestamp=2023-04-13 13:47:43.997 UTC
2024-10-31T15:16:32.540701Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=4550 transaction=4551 timestamp=2023-04-13 15:05:30.690 UTC updated=true elapsed_ms=5.776375
2024-10-31T15:16:32.574600Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1052 transaction=1053 timestamp=2023-04-13 13:48:55.253 UTC
2024-10-31T15:16:32.625726Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1102 transaction=1103 timestamp=2023-04-13 13:49:57.288 UTC
2024-10-31T15:16:32.679135Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1159 transaction=1160 timestamp=2023-04-13 13:51:06.819 UTC
2024-10-31T15:16:32.729858Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1203 transaction=1204 timestamp=2023-04-13 13:52:11.201 UTC
2024-10-31T15:16:32.777044Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1257 transaction=1258 timestamp=2023-04-13 13:53:26.750 UTC
2024-10-31T15:16:32.819603Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1301 transaction=1302 timestamp=2023-04-13 13:54:17.123 UTC
2024-10-31T15:16:32.870028Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1351 transaction=1352 timestamp=2023-04-13 13:55:24.124 UTC
2024-10-31T15:16:32.923059Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1402 transaction=1403 timestamp=2023-04-13 13:56:49.078 UTC
2024-10-31T15:16:32.969159Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1451 transaction=1452 timestamp=2023-04-13 13:57:56.351 UTC
2024-10-31T15:16:33.034831Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="wal_obj_types" epoch=0 checkpoint=5055 transaction=5056 timestamp=2023-04-13 15:16:38.038 UTC updated=true elapsed_ms=1.0442500000000001
2024-10-31T15:16:33.036266Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=1501 transaction=1502 timestamp=2023-04-13 13:58:56.227 UTC
^C2024-10-31T15:16:33.091189Z INFO sui_indexer_alt::pipeline::concurrent::collector: Shutdown received, stopping collector pipeline="wal_obj_types"
2024-10-31T15:16:33.091195Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Shutdown received pipeline="wal_obj_types"
2024-10-31T15:16:33.091201Z INFO sui_indexer_alt::ingestion::regulator: Shutdown received, stopping regulator
2024-10-31T15:16:33.091213Z INFO sui_indexer_alt::metrics: Shutdown received, stopping metrics service
2024-10-31T15:16:33.091225Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Stopping watermark task pipeline="wal_obj_types" watermark=CommitterWatermark { pipeline: "wal_obj_types", epoch_hi_inclusive: 0, checkpoint_hi_inclusive: 5055, tx_hi: 5056, timestamp_ms_hi_inclusive: 1681398998038 }
2024-10-31T15:16:33.091410Z INFO sui_indexer_alt::pipeline::concurrent::committer: Batches done, stopping committer pipeline="wal_obj_types"
2024-10-31T15:16:33.091721Z INFO sui_indexer_alt::pipeline::sequential::committer: Shutdown received pipeline="sum_obj_types"
2024-10-31T15:16:33.091725Z INFO sui_indexer_alt::pipeline::sequential::committer: Stopping committer pipeline="sum_obj_types" watermark=CommitterWatermark { pipeline: "sum_obj_types", epoch_hi_inclusive: 0, checkpoint_hi_inclusive: 1546, tx_hi: 1547, timestamp_ms_hi_inclusive: 1681394391225 }
2024-10-31T15:16:33.093829Z INFO sui_indexer_alt::pipeline::processor: Shutdown received, stopping processor pipeline="wal_obj_types"
2024-10-31T15:16:33.094014Z INFO sui_indexer_alt::pipeline::processor: Shutdown received, stopping processor pipeline="sum_obj_types"
2024-10-31T15:16:33.094406Z INFO sui_indexer_alt::ingestion::broadcaster: Shutdown received, stopping ingestion broadcaster
2024-10-31T15:16:33.094513Z INFO sui_indexer_alt: Indexing pipeline gracefully shut down
```
## Stack
- #20089
- #20114
- #20116
- #20117
- #20119
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 4b727cbbfd89db7ebfd09b21e05faa58ac2d8c96
Author: Ashok Menon <ashok@mystenlabs.com>
Date: Fri Nov 1 00:37:19 2024 +0000
indexer-alt: allow sequential pipeline immediate reset (#20119)
## Description
If the sequential pipeline committer can guarantee that it could process
more checkpoints by looking at its pending buffer, we now reset the
polling interval immediately, so it does not wait to issue the next
write. This mimics a similar behaviour in the concurrent pipeline.
I made this change after noticing how the pipeline behaves when
ingestion is stuck retrying a checkpoint, on my local machine. Usually
when running locally, performance is limited by checkpoint download
rate, but in a sequential pipeline, if a checkpoint failed to download,
it is possible for many checkpoints to end up processed and pending.
With the previous implementation, once ingestion had recovered (the
checkpoint is fetched), the pending buffer kept growing because it was
only able to land `MAX_BATCH_CHECKPOINTS / commit_interval`, so if
checkpoints were getting added faster than that, it would never recover.
With this change, the pipeline recovers almost instantly, and I expect
that in GCP where bandwidth is not the rate limiting factor, this should
improve throughput during backfill, and synthetic benchmarks.
## Test plan
Run the indexer with a large ingestion buffer and concurrency, wait for
ingestion to fail to fetch a checkpoint, and then notice how the
situation recovers (instead of getting worse until the pipeline
eventually complains that it has too many pending checkpoints):
```
sui$ cargo run -p sui-indexer-alt --release -- \
--database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \
indexer --remote-store-url https://checkpoints.mainnet.sui.io \
--last-checkpoint 1200000 --pipeline sum_packages \
--checkpoint-buffer-size 50000 --ingest-concurrency 20000
```
## Stack
- #20089
- #20114
- #20116
- #20117
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit b65f34c6a00801620e11469575523895a8e074ac
Author: Ashok Menon <ashok@mystenlabs.com>
Date: Fri Nov 1 00:21:38 2024 +0000
indexer-alt: wal_coin_balances pipeline (#20117)
## Description
`wal_coin_balances` is to `sum_coin_balances` what `wal_obj_types` is to
`sum_obj_types`.
## Test plan
Run the indexer, and correlate the live object set calculated from the
write-ahead log against the one that's already in the summary table:
```
sui$ cargo run -p sui-indexer-alt --release -- \
--database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \
indexer --remote-store-url https://checkpoints.mainnet.sui.io/ \
--last-checkpoint 5000
```
```
sui_indexer_alt=# SELECT COUNT(*) FROM sum_coin_balances;
count
-------
178
(1 row)
sui_indexer_alt=# SELECT
COUNT(*)
FROM (
SELECT DISTINCT ON (object_id)
*
FROM
wal_coin_balances
ORDER BY
object_id,
object_version DESC
) o
WHERE
o.owner_id IS NOT NULL;
count
-------
178
(1 row)
```
## Stack
- #20089
- #20114
- #20116
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 71f1faa1a8557a178e0bf89f20829a8ea503f156
Author: Ashok Menon <ashok@mystenlabs.com>
Date: Fri Nov 1 00:18:46 2024 +0000
indexer-alt: wal_obj_types pipeline (#20116)
## Description
Adds the concurrent pipeline that writes the write-ahead log for
`sum_obj_types`. It re-uses the `process` implementation from
`sum_obj_types` and then writes it into an append-only table.
Note that today:
- The pipelines are completely independent, which means the WAL pipeline
redoes the processing work of the summary pipeline (this is presumably
not an issue because the repeated work is not particularly heavy).
- This change does not include the pruning necessary to keep this
table's size in check (in practice it should only be a couple of gigs in
size). This will come in a follow-up PR.
## Test plan
Run the indexer and cross check the live-object set calculated from the
write-ahead log with the summary:
```
sui$ cargo run -p sui-indexer-alt --release -- \
--database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \
indexer --remote-store-url https://checkpoints.mainnet.sui.io \
--last-checkpoint 5000
```
```
sui_indexer_alt=# SELECT COUNT(*) FROM sum_obj_types;
count
-------
592
(1 row)
^
sui_indexer_alt=# SELECT
COUNT(*)
FROM (
SELECT DISTINCT ON (object_id)
*
FROM
wal_obj_types
ORDER BY
object_id,
object_version DESC
) o
WHERE
o.owner_kind IS NOT NULL;
count
-------
592
(1 row)
```
## Stack
- #20089
- #20114
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 7de02417c27844c8a086866b1279b4736ab862ec
Author: Ashok Menon <ashok@mystenlabs.com>
Date: Thu Oct 31 23:22:42 2024 +0000
indexer-alt: obj_versions pipeline (#20114)
## Description
Pipeline to fill `obj_versions` table -- this is the table that serves
"parent version" (dynamic field) queries.
## Test plan
```
sui$ cargo run -p sui-indexer-alt --release -- \
--database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \
indexer --remote-url "https://checkpoints.mainnet.sui.io" \
--last-checkpoint 5000
```
## Stack
- #20089
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 53eaed5e300a67f0d01cf68119a145410925f9de
Author: Ashok Menon <ashok@mystenlabs.com>
Date: Wed Oct 30 00:06:07 2024 +0000
indexer-alt: delays for sequential pipelines
## Description
Add the ability to include an optional delay to sequential pipelines.
This can be used to implement behaviour similar to `objects_snapshot` in
the new architecture.
Sequential pipelines honour this configuration in their committer tasks:
- They process checkpoints at the same rate as other pipelines
- But hold back data in the committer's "pending" map until the
committer witnesses a checkpoint that is at least `checkpoint_lag`
checkpoints ahead.
This also requires slightly complicating the set-up and tear-down logic:
- When setting up, we may need to wait until the pending buffer has been
primed enough to set a valid commit upperbound.
- If the pipeline runs dry before that point, we need to add an extra
check to close down the committer.
- When tearing down the committer after its channel has closed, we need
to check whether batching would have picked up the first pending
checkpoint (whether it is a valid "next checkpoint" and it's not held
up because of a lag parameter).
## Test plan
Run the indexer with various parameterisations for the consistent range,
and note that the watermark for the summary pipelines lag behind the
watermarks for other pipelines.
```
sui$ cargo run -p sui-indexer-alt --release -- \
--database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \
indexer --remote-store-url https://checkpoints.mainnet.sui.io \
--pipeline kv_objects \
--pipeline sum_obj_types --pipeline sum_coin_balances \
--last-checkpoint 5000 --consistent-range 1000
[...]
2024-10-30T00:24:45.148259Z INFO sui_indexer_alt::ingestion::regulator: Checkpoints done, stopping regulator
2024-10-30T00:24:45.161682Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=3605 transaction=3606
2024-10-30T00:24:45.212394Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_coin_balances" epoch=0 checkpoint=2703 transaction=2704
2024-10-30T00:24:45.222986Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=3656 transaction=3657
2024-10-30T00:24:45.274318Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=3702 transaction=3703
2024-10-30T00:24:45.340714Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=3753 transaction=3754
2024-10-30T00:24:45.355934Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="kv_objects" elapsed_ms=0.949083 updated=true epoch=0 checkpoint=4760 transaction=4761
2024-10-30T00:24:45.401262Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=3801 transaction=3802
2024-10-30T00:24:45.463134Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=3852 transaction=3853
2024-10-30T00:24:45.544933Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=3904 transaction=3905
2024-10-30T00:24:45.609068Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_obj_types" epoch=0 checkpoint=3952 transaction=3953
2024-10-30T00:24:45.668560Z INFO sui_indexer_alt::ingestion::broadcaster: Checkpoints done, stopping ingestion broadcaster
2024-10-30T00:24:45.668593Z INFO sui_indexer_alt::pipeline::processor: Checkpoints done, stopping processor pipeline="kv_objects"
2024-10-30T00:24:45.668605Z INFO sui_indexer_alt::pipeline::processor: Checkpoints done, stopping processor pipeline="sum_obj_types"
2024-10-30T00:24:45.668605Z INFO sui_indexer_alt::pipeline::processor: Checkpoints done, stopping processor pipeline="sum_coin_balances"
2024-10-30T00:24:45.674305Z INFO sui_indexer_alt::pipeline::sequential::committer: Processor closed channel, pending rows empty, stopping committer pipeline="sum_obj_types" watermark=CommitterWatermark { pipeline: "sum_obj_types", epoch_hi_inclusive: 0, checkpoint_hi_inclusive: 4000, tx_hi: 4001 }
2024-10-30T00:24:45.713856Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_coin_balances" epoch=0 checkpoint=3003 transaction=3004
2024-10-30T00:24:45.863837Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="kv_objects" elapsed_ms=1.83825 updated=true epoch=0 checkpoint=4960 transaction=4961
2024-10-30T00:24:46.120459Z INFO sui_indexer_alt::pipeline::concurrent::collector: Processor closed channel, pending rows empty, stopping collector pipeline="kv_objects"
2024-10-30T00:24:46.131407Z INFO sui_indexer_alt::pipeline::concurrent::committer: Batches done, stopping committer pipeline="kv_objects"
2024-10-30T00:24:46.214267Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_coin_balances" epoch=0 checkpoint=3303 transaction=3304
2024-10-30T00:24:46.356941Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Watermark pipeline="kv_objects" elapsed_ms=0.68 updated=true epoch=0 checkpoint=5000 transaction=5001
2024-10-30T00:24:46.356970Z INFO sui_indexer_alt::pipeline::concurrent::watermark: Committer closed channel, stopping watermark task pipeline="kv_objects" watermark=CommitterWatermark { pipeline: "kv_objects", epoch_hi_inclusive: 0, checkpoint_hi_inclusive: 5000, tx_hi: 5001 }
2024-10-30T00:24:46.714498Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_coin_balances" epoch=0 checkpoint=3603 transaction=3604
2024-10-30T00:24:47.214758Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_coin_balances" epoch=0 checkpoint=3903 transaction=3904
2024-10-30T00:24:47.715841Z INFO sui_indexer_alt::pipeline::sequential::committer: Watermark pipeline="sum_coin_balances" epoch=0 checkpoint=4000 transaction=4001
2024-10-30T00:24:47.715917Z INFO sui_indexer_alt::pipeline::sequential::committer: Processor closed channel, pending rows empty, stopping committer pipeline="sum_coin_balances" watermark=CommitterWatermark { pipeline: "sum_coin_balances", epoch_hi_inclusive: 0, checkpoint_hi_inclusive: 4000, tx_hi: 4001 }
2024-10-30T00:24:47.716485Z INFO sui_indexer_alt: Indexing pipeline gracefully shut down
2024-10-30T00:24:47.716647Z INFO sui_indexer_alt::metrics: Shutdown received, stopping metrics service
```
commit d57326492b62ec8ba7ff8826ad473ec1b5008f11
Author: Ashok Menon <ashok@mystenlabs.com>
Date: Wed Oct 30 00:01:45 2024 +0000
fix(indexer-alt): remove double logging on pipeline skip
## Description
We were logging the pipeline being skipped twice.
## Test plan
Check logs while the indexer starts up:
```
sui$ cargo run -p sui-indexer-alt --release -- \
--database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \
indexer --remote-store-url "https://checkpoints.mainnet.sui.io" \
--last-checkpoint 5000 --pipeline sum_obj_types
```
commit ef0d78c638e3ef0d0a3b28787b618ea56a140284
Author: Ge Gao <106119108+gegaowp@users.noreply.github.com>
Date: Thu Oct 31 19:30:51 2024 -0400
add sui-field-count for sui-indexer-alt (#20126)
## Description
title, so that we can replace hard-coded field count in pipelines.
a crate did very similar things but it's archived, so prob better to
have our own impl, also b/c the functionality is relatively simple
https://github.com/discosultan/field-count
## Test plan
added tests in models/
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit c92dd1b4f16074ced8c1fe5c56a10e675f68bbe6
Author: phoenix <51927076+phoenix-o@users.noreply.github.com>
Date: Thu Oct 31 18:28:34 2024 -0400
[kv store] bigtable. Add support for querying checkpoint by digest (#20122)
## Description
Adds support for querying checkpoint by digest
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 74ace7b6166f10bddfd714568e0ddc20db2ca75d
Author: JasonRUAN <rzexin@gmail.com>
Date: Fri Nov 1 06:25:44 2024 +0800
[CLI] fix `sui keytool import` --alias command and default alias does not work (#20111)
## Description
fix `sui keytool import` --alias command and default alias does not work
Before Fix：
<img width="1295" alt="image"
src="https://github.com/user-attachments/assets/4cb40d92-e266-4b6e-9c39-c05ac52d7a9d">
After Fix：
- **--alias**
<img width="1301" alt="image"
src="https://github.com/user-attachments/assets/b5c711e8-2a4d-4861-9965-30cde260e93f">
- **default alias**
<img width="1295" alt="image"
src="https://github.com/user-attachments/assets/4513db65-0e10-4179-b5f0-de188871cf78">
## Test plan
How did you test the new or updated feature?
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [x] CLI: `sui keytool import --alias` is now fixed and correctly
records the provided alias.
- [ ] Rust SDK:
- [ ] REST API:
commit e41a759769fdea08f660160a23e945bbad80b337
Author: sui-merge-bot[bot] <114704316+sui-merge-bot[bot]@users.noreply.github.com>
Date: Thu Oct 31 23:50:37 2024 +0200
Version Packages (#20127)
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.
# Releases
## @mysten/kms@0.0.3
### Patch Changes
- 02c9e46: Fix exports on the bundled package
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
commit 02c9e4691b429b4a0ec85b71eace37b72515d055
Author: Manolis Liolios <manos@mystenlabs.com>
Date: Thu Oct 31 23:38:15 2024 +0200
Fix build, re-publish @mysten/kms (#20125)
## Description
Build will always treat `src` as the base for outputs when building.
## Test plan
How did you test the new or updated feature?
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 16f2683a99e7b8824aa5879d3f20ffef936a7546
Author: William Smith <williamprincesmith@gmail.com>
Date: Thu Oct 31 17:27:16 2024 -0400
[sui-benchmark] Add expected failure payloads (#19831)
## Description
Introduce a new payload type into the benchmark tool -
`ExpectedFailurePayload`.
This payload type is configured with some expected failure type
(currently only one implemented - user signature failure - but others
can be added with minimal scaffolding) such that
`payload.make_transaction()` generates a transaction that will fail in
this manner. Note that the failures that this payload type is concerned
with are failures to execute the transaction itself, rather than during
execution by the MoveVM. In other words, it is expected that the failure
mode will not consume gas or produce effects.
Note that for this reason, its failure mode is inverted. It will be
tallied for metrics purposes as an `error` if it succeeds and tallied as
an `expected_error` (which is a success) if it fails.
Also note that transaction responses for this type are handled by
producing a `NextOp::Retry` (with some additional logging/metrics) since
it is functionally equivalent to a retryable error.
## Test plan
Ran the following:
```
SIM_STRESS_TEST_QPS=200 SIM_STRESS_TEST_WORKERS=20 RUST_LOG=debug cargo simtest --nocapture test_simulated_load_expected_failure_traffic_control
```
Observed the following in logs
```
2022-01-03T02:05:47.734507Z INFO node{id=1 name="client"}: simtest::test: crates/sui-benchmark/tests/simtest.rs:1088: end of test BenchmarkStats { duration: 50.000002242s, num_error_txes: 0, num_expected_error_txes: 8049, num_success_txes: 348, num_success_cmds: 1977, total_gas_used: 3215776400, latency_ms: HistogramWrapper { histogram: Histogram { auto_resize: false, highest_trackable_value: 120000, lowest_discernible_value: 1, significant_value_digits: 3, bucket_count: 7, sub_bucket_count: 2048, sub_bucket_half_count: 1024, sub_bucket_half_count_magnitude: 10, sub_bucket_mask: 2047, leading_zero_count_base: 53, unit_magnitude: 0, unit_magnitude_mask: 0, max_value: 14359, min_non_zero_value: 72, total_count: 348, counts: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 8, 12, 9, 20, 25, 22, 19
...
```
[Also ran in
PTN](https://metrics.sui.io/d/adl51ctsvmkg0a/traffic-control-dos-protection-dashboard?from=2024-10-31T19:00:50.548Z&to=2024-10-31T20:20:03.673Z&timezone=browser&var-network=private-testnet&var-host=$__all&var-fullnode=$__all)
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit d95918cea0d8fa0f1a4007273f477e3c12b61302
Author: phoenix <51927076+phoenix-o@users.noreply.github.com>
Date: Thu Oct 31 17:18:17 2024 -0400
[data ingestion] add support for bigtable in main workflow binary (#20018)
## Description
adding BigTable KV as a variant to main binary that manages internal
workflows
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit c33374e735257b2a460a2f882e062619f8958bc5
Author: wlmyng <127570466+wlmyng@users.noreply.github.com>
Date: Thu Oct 31 12:51:47 2024 -0700
[mvr] indexer schema subset and reduced persist tasks (#20100)
## Description
Describe the changes or additions included in this PR.
## Test plan
How did you test the new or updated feature?
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 7c517e7e5ce7193d1f51ef7f20a31b56f0e66797
Author: Xun Li <lxfind@gmail.com>
Date: Thu Oct 31 12:49:58 2024 -0700
Add system_invariant_violation macro (#20105)
## Description
Introduce a system invariant violation macro.
We can then set up alerts whenever the counter value changes.
Used it in one place as an example.
## Test plan
How did you test the new or updated feature?
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit b248f35879801b3bae9b76c4d204f36540d816f8
Author: Andrew Schran <aschran@mystenlabs.com>
Date: Thu Oct 31 14:16:48 2024 -0400
Add support for TLS connections with self-signed cert on validator gRPC interface (#19796)
Client side use of TLS will be enabled-by-default in a future PR.
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [x] Nodes (Validators and Full nodes): Adds support for TLS on
validator gRPC interface.
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit e06610823b985bdc01c40e23c9a9a2f8d5b571d1
Author: sui-merge-bot[bot] <114704316+sui-merge-bot[bot]@users.noreply.github.com>
Date: Thu Oct 31 19:58:58 2024 +0200
Version Packages (#20076)
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.
# Releases
## @mysten/sui@1.14.0
### Minor Changes
- c24814b: Adds a custom header; 'Client-Request-Method' which will
contain the method name used in
each outgoing jsonrpc request
## @mysten/create-dapp@0.3.28
### Patch Changes
- Updated dependencies [c24814b]
- @mysten/sui@1.14.0
- @mysten/dapp-kit@0.14.28
## @mysten/dapp-kit@0.14.28
### Patch Changes
- Updated dependencies [c24814b]
- @mysten/sui@1.14.0
- @mysten/wallet-standard@0.13.9
- @mysten/zksend@0.11.9
## @mysten/deepbook@0.8.23
### Patch Changes
- Updated dependencies [c24814b]
- @mysten/sui@1.14.0
## @mysten/deepbook-v3@0.12.2
### Patch Changes
- Updated dependencies [c24814b]
- @mysten/sui@1.14.0
## @mysten/enoki@0.4.7
### Patch Changes
- Updated dependencies [c24814b]
- @mysten/sui@1.14.0
- @mysten/zklogin@0.7.24
## @mysten/graphql-transport@0.2.25
### Patch Changes
- Updated dependencies [c24814b]
- @mysten/sui@1.14.0
## @mysten/kiosk@0.9.23
### Patch Changes
- 4166d71: Fix doc comment on `getKiosk` command
- Updated dependencies [c24814b]
- @mysten/sui@1.14.0
## @mysten/kms@0.0.2
### Patch Changes
- b3f3925: Introduces @mysten/kms which initially exposes a Sui AWS KMS
signer
- Updated dependencies [c24814b]
- @mysten/sui@1.14.0
## @mysten/suins-toolkit@0.5.23
### Patch Changes
- Updated dependencies [c24814b]
- @mysten/sui@1.14.0
## @mysten/wallet-standard@0.13.9
### Patch Changes
- Updated dependencies [c24814b]
- @mysten/sui@1.14.0
## @mysten/zklogin@0.7.24
### Patch Changes
- Updated dependencies [c24814b]
- @mysten/sui@1.14.0
## @mysten/zksend@0.11.9
### Patch Changes
- Updated dependencies [c24814b]
- @mysten/sui@1.14.0
- @mysten/wallet-standard@0.13.9
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
commit 093954a09bce1942a9f9773b845f59bf5124efbf
Author: ronny-mysten <118224482+ronny-mysten@users.noreply.github.com>
Date: Thu Oct 31 11:35:27 2024 -0600
[docs] Rpc update (#20094)
## Description
Updates the RPC render to include component schemas. Also updates a few
comments.
## Test plan
How did you test the new or updated feature?
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit fa66617b7dfa7096fd64a2dc1b002d154223be4a
Author: wlmyng <127570466+wlmyng@users.noreply.github.com>
Date: Thu Oct 31 10:18:57 2024 -0700
[mvr] renaming to sui-mvr-indexer and Dockerfile (#20099)
## Description
Describe the changes or additions included in this PR.
## Test plan
How did you test the new or updated feature?
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 434206f14a0a21bc62ea08480b4a5c4d36d6cc25
Author: Todd Nowacki <tmn@mystenlabs.com>
Date: Thu Oct 31 10:18:46 2024 -0700
[move-compiler] Re-organize warning filters (#20103)
## Description
- Re-organize warning filters. It should make it a bit cleaner to
refactor diagnostic reporting
## Test plan
- ran tests
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 0d16b6f8b7c4ff79f70400e8a94d7486b2b2c3fe
Author: wlmyng <127570466+wlmyng@users.noreply.github.com>
Date: Thu Oct 31 09:36:37 2024 -0700
[mvr] new crate for custom mvr-indexer, copied from sui-indexer crate (#20098)
## Description
Simple copy-paste of the sui-indexer crate for the custom mvr indexer
## Test plan
How did you test the new or updated feature?
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit b3f39253ff0153b69b42bc298d1ecdbb0524e276
Author: John Naulty Jr. <jnaulty@mystenlabs.com>
Date: Thu Oct 31 09:21:29 2024 -0700
AWS KMS TS Signer (#16139)
## Description
Describe the changes or additions included in this PR.
## Test Plan
How did you test the new or updated feature?
---
If your changes are not user-facing and do not break anything, you can
skip the following section. Otherwise, please briefly describe what has
changed under the Release Notes section.
### Type of Change (Check all that apply)
- [ ] protocol change
- [ ] user-visible impact
- [ ] breaking change for a client SDKs
- [ ] breaking change for FNs (FN binary must upgrade)
- [ ] breaking change for validators or node operators (must upgrade
binaries)
- [ ] breaking change for on-chain data layout
- [ ] necessitate either a data wipe or data migration
### Release notes
---------
Co-authored-by: Manolis Liolios <manos@mystenlabs.com>
commit f78c7b006c28c4c1afd3f7d42e21e53970902f5d
Author: Brandon Williams <brandon@mystenlabs.com>
Date: Wed Oct 30 19:50:45 2024 -0500
rest: revert client change to requested protobuf and request bcs instead
Revert the client change to request protobuf as a response type, and
instead request bcs, because the support for protobuf hasn't rolled out
to other networks yet.
commit 108ad4c8ea53c68bd90d0b3fcdc88fbc75640c61
Author: Brandon Williams <brandon@mystenlabs.com>
Date: Wed Oct 30 12:50:43 2024 -0500
rest: support protobuf format for committee endpoints
commit 718a244bca584382faedde56f8247c6b26225b07
Author: Brandon Williams <brandon@mystenlabs.com>
Date: Wed Oct 30 11:03:43 2024 -0500
rest: support protobuf format for resolve, simulate, and execute transaction endpoints
commit 2987f5967ecffa8ddbec2b8ebf9ee2eec09dad60
Author: Brandon Williams <brandon@mystenlabs.com>
Date: Wed Oct 30 08:55:51 2024 -0500
rest: support protobuf format for GetTransaction and ListTransactions endpoints
commit 672ecbd2a58c7514aa24d139c442316a03d7f89e
Author: Brandon Williams <brandon@mystenlabs.com>
Date: Tue Oct 29 21:30:42 2024 -0500
rest: support protobuf format for GetObject endpoints
commit c8472c057523ff88040ae688f1410d34164b0ebc
Author: Brandon Williams <brandon@mystenlabs.com>
Date: Tue Oct 29 19:00:39 2024 -0500
rest: remove ListFullCheckpoint api
Remove the ListFullCheckpoint api as its a bit too heavy-weight and
instead we'll look to introduce a streaming api in the future.
commit 39f1ae21fd5467e71522729519720d2add9468dc
Author: Xun Li <lxfind@gmail.com>
Date: Wed Oct 30 22:25:50 2024 -0700
Clean up local execution (#20086)
## Description
Describe the changes or additions included in this PR.
## Test plan
How did you test the new or updated feature?
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 65b02e203ad897f438f87d0e5cfe46c56a1d4247
Author: phoenix <51927076+phoenix-o@users.noreply.github.com>
Date: Wed Oct 30 21:26:54 2024 -0400
[pruner] enable periodic compaction of individual SST files by default (#20101)
## Description
enables periodic compaction of individual SST files by default
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit f2670718f694edd59e8a062bc932d4efd96ba765
Author: Tim Zakian <2895723+tzakian@users.noreply.github.com>
Date: Wed Oct 30 16:49:01 2024 -0700
[execution] Update to deps-only mode (#20102)
## Description
Disallow new modules from being added to packages in deps-only mode.
## Test plan
Added new tests to check for both previous and new behavior.
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [X] Protocol: Updated restrictions on deps-only packages to not allow
adding new modules to the package.
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
commit 7e6319fa33b7021a94ff7dcc07e479a4bae92b55
Author: Adam Welc <adam@mystenlabs.com>
Date: Wed Oct 30 15:51:37 2024 -0700
[trace-view] Added support for native functions and global locations (#20080)
## Description
Adds support for native functions where the `CloseFrame` event happens
right after `OpenFrame` event and both need to be skipped to skip over
the native function. Also added support for handling global locations -
they should be parsed properly but largely ignored in the trace as they
cannot directly affect values of local variables
## Test plan
All old and new tests must pass
commit 21639a51847f1a04b6b221ac77df3875c0430be7
Author: Xun Li <lxfind@gmail.com>
Date: Wed Oct 30 14:30:59 2024 -0700
[RPC] Fix coin metadata api inefficiency (#19794)
## Description
This PR fixes a minor inefficiency in the coin metadata api.
It does not need to fetch the transaction, as it only needs the effects.
This likely avoids a remote KV fetch.
## Test plan
CI
---
## Release notes
Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.
For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.
- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:
comm…
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Disallow new modules from being added to packages in deps-only mode.
Test plan
Added new tests to check for both previous and new behavior.
Release notes
Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.
For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.