[Sharded-Execution-GRPC] Add GRPC communication for sharded execution. #10274

manudhundi · 2023-09-27T19:06:44Z

Description

In this commit we replace the existing socket based communication with GRPC.
Here we get the basic GRPC reliability.

More reliability and better performance to come in subsequent commits

Test Plan

sitalkedia

Great work. Looks good overall % minor nits.

sitalkedia · 2023-09-27T20:56:13Z

execution/executor-service/src/tests.rs

+
+    // wait for the servers to be ready before sending messages
+    // TODO: We need to pass this test without this sleep
+    thread::sleep(std::time::Duration::from_millis(10));


What is needed to address this TODO? Possible to address this in this PR?

wait for server to start.. Yes will address

sitalkedia · 2023-09-27T20:57:32Z

secure/net/src/grpc_network_service/mod.rs

+    Request, Response, Status,
+};
+
+pub struct RemoteExecutionServerWrapper {


This is generic and nothing to do with the execution server.. May be a better naming would be gRPCNetworkServer ?

sitalkedia · 2023-09-27T20:59:53Z

secure/net/src/grpc_network_service/mod.rs

+    ) {
+        rt.spawn(async move {
+            self.start_async(server_addr, server_shutdown_rx).await;
+        });


Don't you need to wait for server to start here? This can be done by creating a one_shot channel and sending a signal on that channel after self.start_async(server_addr, server_shutdown_rx).await; and waiting to receive on that channel before returning from start .

Yes, one_shot channel is a good idea.. This is basically my 3 of my TODOs. Waiting for server to start before sending messages. Hence those sleeps in test cases

There is no easy way to do this by APIs provided by GRPC or Tonic. Ways to achieve this would be by doing a check/retry from client side.
As discussed going to leave this a TODO for now

One way to do this would be implementing a healthcheck service
https://stackoverflow.com/questions/66798278/golang-grpc-how-to-determine-when-the-server-has-started-listening

Health check service is a good idea.

Maybe we can just use the reflection service?

That is a good point, I will check into it

sitalkedia · 2023-09-27T21:01:43Z

crates/aptos-protos/proto/aptos/remote_executor/v1/network_msg.proto

+
+}
+
+service RemoteExecution {


The name should be generic.. something like NetworkingService

I will use 'NetworkMessageService'

sitalkedia · 2023-09-27T21:03:03Z

secure/net/src/grpc_network_service/mod.rs

+        // TODO: Retry with exponential backoff on failure
+        match self.remote_channel.simple_msg_exchange(request).await {
+            Ok(_) => {},
+            Err(e) => {


Hmm we should just panic in this case, right?

Ideally we should have retries. There could be temporary failures on server. We can panic for now.

You can add some retry with exponetial back-off I assume..and panic once we exhaust the retries..

Adding retries requires an additional copy of msg to send.
(This can be achieved by existing retry_async() library)

Since we do not want to bite the bullet of the cost of additional copy, leaving retry as a TODO for now

sitalkedia · 2023-09-27T21:04:00Z

secure/net/src/network_controller/inbound_handler.rs

    inbound_handlers: Arc<Mutex<HashMap<MessageType, Sender<Message>>>>,
 }

 impl InboundHandler {
-    pub fn new(service: String, listen_addr: SocketAddr, timeout_ms: u64) -> Self {
+    pub fn new(service: String, listen_addr: SocketAddr, _: u64) -> Self {


remove the timeout if not applicable anymore.. how do we configure the timeout for gRPC btw?

Using the timeouts as is. We can set in gRPC when we create the server

secure/net/src/network_controller/mod.rs

sitalkedia · 2023-09-27T21:12:03Z

secure/net/src/network_controller/outbound_handler.rs

+        // the cost of the mutex when there is no contention
+        let outbound_handlers = mem::take(self.handlers.as_mut());
+
+        // TODO: Consider using multiple tasks for outbound handlers


What would it take to multi-thread this? I am afraid, without processing messages in multiple threads, this will be a chokepoint in remote execution environment.

I suppose minimal overhead. Just the logic of splitting the outbound_handlers across the threads/tasks

sitalkedia · 2023-09-27T21:13:22Z

secure/net/src/network_controller/outbound_handler.rs

+                    },
+                    Err(e) => {
+                        // not necessarily an error, just means a sender closed a channel
+                        trace!(


Why is this not an error? Silently ignoring error here might mean the transport layer doesn't provide reliable message delivery..

This just means one of the senders has closed the send channel. It can happen when the sender_tx channel is dropped after it has done its job.
Also, we should not panic at this level if one the sender dies because many senders subscribe to this outbound task. If a sender_tx channel dies due to an error, then it is application error, not transport layer error.

This can mean many things (not necessarily sender dropping).. so I would be hesitant to ignore this and continue.. what is the downside of panic/exiting from this loop when this happens?

Ok, logging the a warning here.

@manudhundi - As discussed, instead of continuing here, let's either panic or shut down the handler in case of error?

Sorry, my comment was not clear. Shutting down the handler indeed (after logging the warning). Panic and shutdown would mean the same I suppose as the task just runs this handler

In case we want to shut down, we should break out of this loop right?

Yes we do break out using the signal from the msg of type "stop_task"

crates/aptos-protos/proto/aptos/remote_executor/v1/network_msg.proto

crates/aptos-protos/src/pb/aptos.remote_executor.v1.rs

crates/aptos-protos/proto/aptos/remote_executor/v1/network_msg.proto

sitalkedia · 2023-09-29T17:37:48Z

secure/net/src/grpc_network_service/mod.rs

+    ) {
+        rt.spawn(async move {
+            self.start_async(server_addr, server_shutdown_rx).await;
+        });


Health check service is a good idea.

sitalkedia · 2023-09-29T17:41:18Z

secure/net/src/network_controller/outbound_handler.rs

+
+        // Register a signal handler to stop the outbound task
+        let (stop_signal_tx, stop_signal_rx) = unbounded();
+        self.handlers.push((


Better you do it in the constructor and remove the handler.is_empty() check on line 53.

That will require all the handlers registered in the constructor ? Will that not break the nice modularity that we have ? That is 1 network controller/outbound_task per node. And many modules are individually register their handlers

sitalkedia · 2023-09-29T17:42:02Z

secure/net/src/network_controller/outbound_handler.rs


-        retry(fixed_retry_strategy(5, 20), || network_client.write(&msg)).unwrap();
+            if message_type.get_type() == "stop_task" {


What is the use of this stop message?

It is a clean way of exiting this task. NetworkController::shutdown calls this

hmm but why do we need an explicit shutdown? When the run time goes out of scope, I assume it should auto shutdown?

For 2 reasons:
(1) Runtime going out scope kills the task in a heavy handed way. That is, if the task in middle of something, it will die in between.
(2) Not an expert on tokio runtimes, but i have seen hangs sometime. That is, runtime object is not dropped until the task dies

(1) Runtime going out scope kills the task in a heavy handed way. That is, if the task in middle of something, it will die in between.

This shouldn't be a problem right as the run time going out of scope means we are shutting down the controller anyways..

Do you see the test hanging without the shutdown support?

sitalkedia

Stamping to unblock, there are few TODOs like error handling, shutdown mechanism etc that we need to follow up.

manudhundi · 2023-10-02T20:53:23Z

Thanks. Yup, will follow up on the shutdown mechanism and other TODOs once we have some reasonable perf numbers with remote sharding

msmouse · 2023-10-03T16:49:28Z

execution/executor-service/src/remote_state_view.rs

+                // Though we try to insert the state key during the 'pre_fetch_state_values' remote
+                // request, there is a possibility that the state key is not inserted before we
+                // receive and process the response here. To be on the safe side, we insert the
+                // state key here if it is not inserted already. This does not matter because we
+                // do not have versioning of the state view.
+                state_view_lock.insert_state_key(state_key.clone());


still needed?

Not needed, I will remove it.. I will rebase on previous change

msmouse · 2023-10-03T17:36:00Z

secure/net/src/grpc_network_service/mod.rs

+    ) {
+        rt.spawn(async move {
+            self.start_async(server_addr, server_shutdown_rx).await;
+        });


Maybe we can just use the reflection service?

In this commit we replace the existing socket based communication with GRPC. Here we get the basic GRPC reliability. More reliability and better performance to come in subsequent commits

github-actions · 2023-10-03T22:04:00Z

✅ Forge suite `compat` success on `aptos-node-v1.6.2` ==> `610363e5a86578196cdfd6519d9ffb06a5bf1cb8`

Compatibility test results for aptos-node-v1.6.2 ==> 610363e5a86578196cdfd6519d9ffb06a5bf1cb8 (PR)
1. Check liveness of validators at old version: aptos-node-v1.6.2
compatibility::simple-validator-upgrade::liveness-check : committed: 3874 txn/s, latency: 8134 ms, (p50: 8700 ms, p90: 10500 ms, p99: 12400 ms), latency samples: 151100
2. Upgrading first Validator to new version: 610363e5a86578196cdfd6519d9ffb06a5bf1cb8
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1775 txn/s, latency: 16276 ms, (p50: 18700 ms, p90: 22200 ms, p99: 22500 ms), latency samples: 92320
3. Upgrading rest of first batch to new version: 610363e5a86578196cdfd6519d9ffb06a5bf1cb8
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1111 txn/s, latency: 16394 ms, (p50: 19000 ms, p90: 22500 ms, p99: 23400 ms), latency samples: 91120
4. upgrading second batch to new version: 610363e5a86578196cdfd6519d9ffb06a5bf1cb8
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 2960 txn/s, latency: 9152 ms, (p50: 9600 ms, p90: 12000 ms, p99: 40000 ms), latency samples: 139120
5. check swarm health
Compatibility test for aptos-node-v1.6.2 ==> 610363e5a86578196cdfd6519d9ffb06a5bf1cb8 passed
Test Ok

github-actions · 2023-10-03T22:05:00Z

✅ Forge suite `realistic_env_max_load` success on `610363e5a86578196cdfd6519d9ffb06a5bf1cb8`

two traffics test: inner traffic : committed: 5985 txn/s, latency: 6535 ms, (p50: 6300 ms, p90: 8100 ms, p99: 15300 ms), latency samples: 2597700
two traffics test : committed: 100 txn/s, latency: 2441 ms, (p50: 2300 ms, p90: 2800 ms, p99: 5000 ms), latency samples: 1820
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.207, avg: 0.198", "QsPosToProposal: max: 0.396, avg: 0.192", "ConsensusProposalToOrdered: max: 0.671, avg: 0.637", "ConsensusOrderedToCommit: max: 0.563, avg: 0.532", "ConsensusProposalToCommit: max: 1.220, avg: 1.168"]
Max round gap was 1 [limit 4] at version 1292854. Max no progress secs was 5.189009 [limit 10] at version 1292854.
Test Ok

github-actions · 2023-10-03T22:22:07Z

✅ Forge suite `framework_upgrade` success on `aptos-node-v1.5.1` ==> `610363e5a86578196cdfd6519d9ffb06a5bf1cb8`

Compatibility test results for aptos-node-v1.5.1 ==> 610363e5a86578196cdfd6519d9ffb06a5bf1cb8 (PR)
Upgrade the nodes to version: 610363e5a86578196cdfd6519d9ffb06a5bf1cb8
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 4145 txn/s, latency: 5060 ms, (p50: 5100 ms, p90: 8100 ms, p99: 8700 ms), latency samples: 232120
5. check swarm health
Compatibility test for aptos-node-v1.5.1 ==> 610363e5a86578196cdfd6519d9ffb06a5bf1cb8 passed
Test Ok

aptos-labs#10274) [Sharded-Execution-GRPC] Add GRPC communication for sharded execution In this commit we replace the existing socket based communication (that is message send and message recv) with GRPC. Here we get the basic GRPC reliability. More reliability and better performance to come in subsequent commits

@main

* Update system-integrators-guide.md (aptos-labs#10087) The old url file doesn't exist anymore. must be changed to the new one. * [dashboards] sync grafana dashboards * [typo*] (aptos-labs#9912) * [typo*]Update delegation-pool-operations.md * [typo*]Update run-a-fullnode-on-gcp.md * [typo*]Update run-a-fullnode-on-gcp.md * [typo*]Update glossary.md * [Spec] Ensures for stake.move (aptos-labs#9700) * 1 * cannot finish hp * remove some wrong statements * hp1-3 * rewrite hp2 * rewrite hp2 again * hp1-3 * init * fix * fix ensures * fix ensure in the update_sat * fix comment * fix md * fix new comment * update comment * fix indent * fix timeout * fix timeout * fix timeout --------- Co-authored-by: chan-bing <zzywullr@gmail.com> * Update fullnode-source-code-or-docker.md (aptos-labs#9914) * [typo*]Update index.md (aptos-labs#9913) * [typo*]Update index.md * Update index.md --------- Co-authored-by: Christian Sahar <125399153+saharct@users.noreply.github.com> * [link*Update fullnode-source-code-or-docker.md (aptos-labs#9978) * Update aptos-bitvec/src/lib.rs (aptos-labs#10045) There is a mistake in the comment * Update delegation-pool-operations.md (aptos-labs#10171) remove deprecated function * trivial: fix rocksdb property reporter * forge: ability to override resource ask in Rust * [CLI] Add txn stream to local testnet (aptos-labs#10101) * [dashboards] sync grafana dashboards * [TS SDK V2] Add `General` api class (aptos-labs#10185) * add general api queries * add general api queries * [move] make vm clonable * Pass down resolver to MoveVmExt::new() In preparation for caching a warm VM * Move VM warm up into adapter * WarmVmCache * Fix circular dependency * [object] refactor burn/unburn (aptos-labs#9785) * Fix Issue 9717 by breaking critical edges in CFG in move-compiler (v1) (aptos-labs#10064) Fix issue aptos-labs#9717 by breaking critical edges in the CFG in move-compiler (v1). While issue 9717 mentions "inline", this is a red herring as inlining a vector loop just makes it more likely to have a reference parameter on the stack, which leads to a failure to correctly drop dead references in move-compiler/src/cfgir/liveness/mod.rs. I've left the initial test case bug_9717.move and added bug_9717_looponly.move that just has a loop illustrating the problem, along with several variants on the loop (one of them, break2, also exhibiting the problem). Anyway, the failure to properly place the reference drop happens when the constructed CFG has critical edges (edges from a node with multiple outgoing edges to a node with multiple ingoing edges). Such a situation is well-known in compiler literature to make it difficult to place instructions precisely based on certain analyses. Fortunately, this seems to be easily fixed by adding a small pass to add a node on each such critical edge, so that the drop can be placed properly even in the case of a break. Unfortunately, later passes can't deal with the resulting deep expression trees and chains of direct jumps that result in some cases, so I had to fix hlir/translate.rs to reduce stack depth for a given expression, and cfgir/optimize/inline_blocks.rs to properly remove the unneeded jumps. New tests also reveal misplaced warnings/errors about unused variables in the presence of inlining, so that was also fixed. * [Doc] Update build e2e dapp docs (aptos-labs#10165) * Update build e2e dapp docs * update js -> jsx * update * Update developer-docs-site/docs/tutorials/build-e2e-dapp/4-fetch-data-from-chain.md * f * f --------- Co-authored-by: David Wolinsky <isaac.wolinsky@gmail.com> * implement memoize high order function (aptos-labs#10110) * [Executor] Metadata and exists support (in Block-STM/executor) (aptos-labs#10170) * [move-vm] Refactor Loader (aptos-labs#9320) * Rename variant * [move-vm] Store ability in runtime type * fixup! [move-vm] Store ability in runtime type * fixup! fixup! [move-vm] Store ability in runtime type * fixup! fixup! fixup! [move-vm] Store ability in runtime type * [move_vm] Split loader into multiple files * [move_vm] Cache struct type for modules * [move_vm] Move depth cache to type cache * [move-vm] Use name to replace type index * [move-vm] Remove function index * [move-vm] Inline functions * [move-vm] Remove cached index from definition * [move-vm] Check cross module linking before loading * [move-vm] Remove global struct cache * [move-vm] Inline struct name to avoid excessive memory allocation * [move-vm] Split function in loader * [move-vm] Split out module cache * [e2e-test] Add randomized test for loader * Fix Lint * [move-vm] Cache signature resolution * [move-vm] Removed unneeded signature token * [move-vm] Arc-ed type argument * Fix Zekun's comments * fixup! [e2e-test] Add randomized test for loader * add comments for the test strategy * Rename struct name * More renaming * Addressing more comments * [consensus] Dedicated channel for proposal buffering and batch process * revert 9ed1da8 (aptos-labs#10256) * update move run to include more params (aptos-labs#9932) * [CLI] Update CLI binary doc site to include openssl3 instruction (aptos-labs#9964) * Update CLI binary doc site to include openssl3 instruction * Update developer-docs-site/docs/tools/aptos-cli/install-cli/download-cli-binaries.md Co-authored-by: Christian Sahar <125399153+saharct@users.noreply.github.com> * Update developer-docs-site/docs/tools/aptos-cli/install-cli/download-cli-binaries.md Co-authored-by: Christian Sahar <125399153+saharct@users.noreply.github.com> --------- Co-authored-by: David Wolinsky <isaac.wolinsky@gmail.com> Co-authored-by: Christian Sahar <125399153+saharct@users.noreply.github.com> * [dag] split notifier into Order and Proof Notifier [dag] additional ledger info verification checks [dag] separate out highest committed round provider [dag] introduce a ledger info provider trait * [CLI] Fix faucet component of local testnet without --force-restart (aptos-labs#10214) * Update Docker images (aptos-labs#10208) Co-authored-by: gedigi <gedigi@users.noreply.github.com> * [ts-sdk-v2] Add .npmrc to ensure pnpm lockfile is consistent (aptos-labs#10251) * Update dependabot.yml to disable version update PRs (aptos-labs#10249) * CLI version bump to 2.1.1 (aptos-labs#10272) * Get Drand Move example to compile (aptos-labs#10196) * change names of drand and veiledcoin * drand compiles * drand tests compile * remove diff.txt * add retrieve lottery winner function * Update CLI dry run text (aptos-labs#10273) * add partial transaction queries (aptos-labs#10275) * [TSSDKv2][1/n] add Ed25519 classes (aptos-labs#10157) * add PrivateKey, PublicKey, and Signature classes * Add helper.ts and fixes based on comments * add asymmetric_crypto file which includes all crypto base classes as abstract. And have all concrete crypto classes to extend from based abstract classes * Update crypto classes name to include Ed25519 prefix * [TS SDK V2] Add Indexer account queries (aptos-labs#10216) * implement account indexer queries * add indexer account api queries * address feedback * [TSSDKv2][3/n] Account and AuthenticationKey Classes (aptos-labs#10210) * Add helper.ts and fixes based on comments * add asymmetric_crypto file which includes all crypto base classes as abstract. And have all concrete crypto classes to extend from based abstract classes * Update crypto classes name to include Ed25519 prefix * fixes based on comments * Add account and auth key classes * update Account class to accept abstract PublicKey and PrivateKey * update methods' comment * [ts-sdk-v2] Add getTransactions to the sdk-v2 transaction API (aptos-labs#10288) * Add getTransactions to the sdkv2 transaction API * Update ecosystem/typescript/sdk_v2/src/internal/transaction.ts Co-authored-by: Maayan <maayan@aptoslabs.com> * update after merge --------- Co-authored-by: Greg Nazario <greg@gnazar.io> Co-authored-by: Maayan <maayan@aptoslabs.com> * [aptos-stdlib] Cleanup error codes for divide by 0 * [marketplace-example] Fix royalties edge conditions There were two bugs with royalties and listings. One was that in v1, royalties could have a denominator 0, or be greater than 100% in legacy NFTs. This means that it's possible that payouts don't work correctly, or take more than expected. Now, royalties are bounded to 0-100%. The second issue was that commission was taken after royalties, but didn't consider that royalties could be 100%. Now, royalties are taken first, and commission is taken out of the remainder. This does mean the marketplace may not have any commission if the royalties are set to 100%. * [Spec] Fix spec (aptos-labs#10215) * fix spec * staking_contract spec * [Rust] Upgrade to Rust version 1.72.1 * [Forge][Chaos] remove jitter, make inter-region BW 300 Mbps (aptos-labs#10277) ### Description The changes are based on observations while measuring network performance and reading more into the dataset used. * Jitter: My understanding is that re-ordering of packets should be pretty rare in the real world, while the previous jitter configs would introduce re-ordering quite frequently. Unless we have a strong belief that jitter is present in our networks, we shouldn't mess with this. * Inter-region bitrate: The numbers this was based on are iperf with a single TCP stream. The results correlate strongly with RTT, which suggests that RTT is the limiting factor, so there's no real reason to constrain BW itself. For now, 300 Mbps is as fast as our network stack will go for 100+ ms RTT. * [Forge][PFN] remove epoch changes from some tests (aptos-labs#10278) ### Description This is in an effort to reduce noise in the PFN tests. Epoch changes are an obvious source of noise, although it's unclear how much it contributes to noise difference between runs. To still have coverage of epoch changes, the tests without chaos will still do 5 minute epoch changes. ### Test Plan Run ad-hoc forge run, observe no epoch changes. * [Forge][netbench] split into large and small messages for two region test (aptos-labs#10283) ### Description We found that the throughput for large and small messages is very different with large latencies. We update the test to run with large messages and then split out a small messages test. The large messages are useful for sanity checking the network setup, the small messages is something that we can hopefully improve upon. * [crypto] add secp256k1 support to aptos-crypto Pretty straight forward, but I think our crypto apis are really not great. The amount of traits that should really be compressed down into a single trait for PublicKey, PrivateKey, and Signature. This would make maintaining and adding new libraries so much easier. * [types] Add support for secp256k1 authenticator with this we can now send secp256k1 signed transactions to the blockchain... I'm going to do some code refactoring in authenticators and transactions before resuming the end-to-end testing and the feature gating of this feature. * [openapi] update openapi with secp256k1 * [types] remove AuthenticationKeyPreimage This was adding extra code and adding complexity to what is already a complex space. If we aren't going to use the preimage, we have no need to write the code. We need to be more dilligent about removing this type of unnecessary code from the codebase, because it really impairs our ability to move fast. * [types] remove prefix and rename derived_address on AuthenticationKey * AuthenticationKey and Address are 1:1, so all this code is legacy based upon some weird goal of trying to compress the account address into an insecure size back in Libra. Even the authors of this code have since moved to 32-bytes in their own blockchain. * prefix is never used and removed. * derived_address -> account_address because there's no derivation it is literally 1:1 * [api] end to end test for secp256k1 ecdsa * [features] add secp256k1 ecdsa * [Aptos Data Poller] Improve peer polling logic. * [Aptos Data Client] Update tests for new poller. * [exp] make it work for any upstream repo (aptos-labs#10016) * [indexer-grpc] k6 loadtest (aptos-labs#9493) * [Aptos Data Client] Improve selection for optimistic fetch and subscriptions. * [Aptos Data Client] Add new tests for peer selection logic. * Make InMemoryStateCalculatorV2 work with state sync chunks. (aptos-labs#10263) * [release-builder] Increase lockup before executing proposals Increase the lockup before executing proposals Currently the release flow started consistently failing because we need to increase the lockup This ensures that the lockup is sufficient before executing transactions Test Plan: not sure how to test other than by running against testnet??? Please advise * [Network] Replace RwLock with ArcSwap for trusted peers. * [Network] Reduce lock contention for peer metadata using cache. * replay-verify.yaml not reference workflows by @main * replay-verify: not cancel other sub-jobs on first failure * recalibrate single node benchmark for perf regression aptos-labs#10298 * [move-model] fix internal assertion violation in definition analysis (aptos-labs#10292) Co-authored-by: Aalok Thakkar <aalok@Aaloks-MacBook-Pro.local> * [ts-sdk-v2] Rename endpoint to path in client * [ts-sdk-v2] Rename originMethod to name for API requests * [ts-sdk-v2] Simplify fullnode get requests * [ts-sdk-v2] Make Post requests simpler * [ts-sdk-v2] Update format and lint for SDK This fixes most of the SDK lints except for the pieces around the static functions for deserialize. * [ts-sdk-v2] Fix broken tests * [ts-sdk-v2] Replace "Generic error" * [ts-sdk-v2] Add derivation path invalid test * [ts-sdk-v2] Cleanup Account and crypto for code reuse 1. Allows deriving public key from private key 2. Detaches accounts from Single Ed25519 3. Adds some consistency to input naming * [ts-sdk-v2] Unify authentication key creation * [ts-sdk-v2] Add Authentication Key scheme enum * [ts-sdk-v2] Add ability to derive authkeys from non-key schemes * [ts-sdk-v2] Add docs to AuthenticationKey * [ts-sdk-v2] Cleanup documentation on Ed25519 * [ts-sdk-v2] Add documentation to asymmetric crypto * [ts-sdk-v2] Add docs, cleanup multi-ed25519 * [ts-sdk-v2] Move paginate with cursor to client * [ts-sdk-v2] Cleanup unused lint ignores * [ts-sdk-v2] Authentication key testing and message improvements * [ts-sdk-v2] Rename some client inputs and add documentation * [dag] epoch manager integration; dag is here * [dashboards] sync grafana dashboards * fix WarmVmCache Natives are stateful so must be covered by the WarmVmId. 1. add TimedFeaturesBuilder and convert TimedFeatures to an array of booleans. 2. add SafeNativeBulder::id_bytes() to fix the bug 3. rebuild vm (and hence all the natives) only when the builder id_bytes() change. * use s5cmd for downloading files in replay-verify official aswcli experiencing random crashes * [e2e-tests] Fix compilation error (aptos-labs#10318) * Drop frozen root after make checkpoint. (aptos-labs#10327) * redistribute mainnet replay sub-job ranges * [TS-SDK v2] Updating the `Deserializable` interface and making `Serializable` an abstract class (aptos-labs#10307) * Removing export from Deserializable to facilitate using a static `deserialize` method, fixing error messages in unit tests for multi_ed25519, and changing Serializable to an abstract class that implements `bcsToBytes()`. Removed abstract deserialize from public/private key classes. * Re-adding doc comments * Adding `serialize` and `deserialize` and corresponding unit tests to the AccountAddress class * Fixing multi_ed25519 error messages * Updating doc comments for Deserializable to clarify what its purpose is. * [move unit tests] Run extended checker as part of unit tests (aptos-labs#10309) * [move unit tests] Run extended checker as part of unit tests Closes aptos-labs#9251 This runs the extended checker as part of Aptos unit tests (either our own Rust integrated tests or from the CLI). It uses the same technique as we already used for native extensions specific to Aptos: a hook is defined where additional, move-model based validations can be run. This is hook is then connected to the extended checker when running Aptos tests. The implementation also optimizes the construction of the move model: if that one is already needed by abi generation (which is the default), it is not constructed a 2nd time for the extended checker -- both for the existing build step and the new test step. This should avoid one full additional compilation (source -> bytecode -> model run). * Extended checks until now excluded test code, leading to wrong usage of entry functions and attributes marked as test-only. Because fixing this is a breaking change, this commit adds the behavior to check test code via a new CLI option `--check-test-code`. This flag should eventually become default behavior. Also fixes some reviewer comments. * implement forge links to axiom (aptos-labs#10330) * [gas-calibration][simple] ignore terms with 0-coefficients (aptos-labs#9742) * [indexer][api] update the metrcis for api gateway consumption. (aptos-labs#10322) * [dag] add various counters * [TS-SDK v2] Adding `serializeVector` and `deserializeVector` to SDK v2 (aptos-labs#10347) * Adding `serializeVector` and `deserializeVector` to serializer.ts and deserializer.ts as well as a unit test for each * Adding documentation for each function * Removing redundant line in doc comment * Removing redundant line in deserializer doc comment too * Allow using skip_index_and_usage on state sync code path. (aptos-labs#10303) * [dag] add a few structured logging * jin_fix_ed25519_derive_publickey (aptos-labs#10357) * [ts-sdk-v2] Add MIME types, and convert input types from string to Enum (aptos-labs#10308) * Remove expensive counters (aptos-labs#10188) * [TS SDK V2] Port over transaction types (aptos-labs#10364) * transaction types * address comments * [CLI] Restructure local testnet code (aptos-labs#10252) * [release-builder] Fix increase lockup The previous PR was converting the private key arg to string incorrectly... ED25519 key type has a very silly to_string that returns debug output I suppose this is to prevent dumping the key, but really i think it should just not have a to_string and instead only have a debug method that does the same thing, and a very explicit to hex method so that you dont accidentally dump these. Test Plan: framework upgrade test succeeds, or at least doesnt fail on this error (there is still some underlying flakiness in the forge test itself) * [Sharded-Execution] Fix a race condition while fetching the state values on a shard from a remote stateview (aptos-labs#10320) [Sharded-Execution] Fix a race condition while fetching the state values on a shard from a remote stateview * [SDKv2] derive publickey unit test (aptos-labs#10358) * jin_fix_ed25519_derive_publickey * Add unit test for publicKey() derivate method * [framework] Turn on test-only checking and fix errors (aptos-labs#10368) This turns on running extended checks also on test-only code in the framework unit tests. A few errors discovered this way are fixed. Changes to function declarations in this PR do not effect compatibility because only test-only functions are effected which are stripped before deployment. Related to aptos-labs#10335, but more needs to be done to make this behavior the default. This cannot happen before the next framework release. * Add CI to check for banned CLI deps (aptos-labs#10338) * refactor: replace multiply_then_divide using math64::mul_div (aptos-labs#10047) Co-authored-by: Kevin <105028215+movekevin@users.noreply.github.com> * [dag] hardening message verification * [TS SDK V2] Port over account and transaction authenticator, signed transaction type (aptos-labs#10367) * transaction types * address comments * authenticators * previewnet flow in the benchmark (aptos-labs#10305) * [ts sdk-v2] Mini get/post request refactor (aptos-labs#10380) * [Sharded-Execution-GRPC] Add GRPC communication for sharded execution. (aptos-labs#10274) [Sharded-Execution-GRPC] Add GRPC communication for sharded execution In this commit we replace the existing socket based communication (that is message send and message recv) with GRPC. Here we get the basic GRPC reliability. More reliability and better performance to come in subsequent commits * [forge] Fix fullnode override in forge (aptos-labs#10382) ### Description Fixing a mistake made in a previous PR, that made fullnode configs not apply (and overwrite validator configs) ### Test Plan Ran a fullnode test and manually check the config that is logged. * Update delegation-pool-operations.md (aptos-labs#10299) reorganize page * [indexer grpc] update the metrics for usage analysis. (aptos-labs#10383) * Update staking-pool-operations.md (aptos-labs#10300) * Update staking-pool-operations.md reorganize page * Update staking-pool-operations.md add step for owner account * Update staking-pool-operations.md * Update staking-pool-operations.md remove heading --------- Co-authored-by: Christian Sahar <125399153+saharct@users.noreply.github.com> * [Storage][Pruner] Set min_readable_version and metrics. (aptos-labs#10381) * rename enum variants and transaction arguments (aptos-labs#10384) * [indexer grpc] More fields to short connection metric (aptos-labs#10385) * [Tutorials] Replacing the old "Your First NFT" tutorial entry function calls to use aptos_token.move and new indexer queries (aptos-labs#9424) * Overwriting the old your-first-nft tutorial in the docs to replace the 0x3 contract calls with the aptos_token contract calls. Also added corresponding documentation tags to referenced indexer queries and the new token client calls. * Changing simple-aptos-token.py to simple_aptos_token.py and updating the Makefile to include it as an example * Changing `create_token` to `mint_token` in python tutorial and `createToken` in typescript to `mint` * Updating typescript example to work correctly if indexer/fullnode chainId aren't in sync * [Python SDK] Use AccountAddress.from_str_relaxed when parsing addresses from the node API --------- Co-authored-by: rtmtree <36580295+rtmtree@users.noreply.github.com> Co-authored-by: rustielin <rustielin@users.noreply.github.com> Co-authored-by: Vladislav ~ cryptomolot <88001005+cryptomolot@users.noreply.github.com> Co-authored-by: Zorrot Chen <UI_Zorrot@163.com> Co-authored-by: chan-bing <zzywullr@gmail.com> Co-authored-by: Freesson ~ cryptomolot <semenikhinas@gmail.com> Co-authored-by: Christian Sahar <125399153+saharct@users.noreply.github.com> Co-authored-by: Jiege <stevekeol.x@gmail.com> Co-authored-by: michelle-aptos <120680608+michelle-aptos@users.noreply.github.com> Co-authored-by: aldenhu <msmouse@gmail.com> Co-authored-by: Daniel Porteous (dport) <daniel@dport.me> Co-authored-by: Maayan <maayan@aptoslabs.com> Co-authored-by: Zekun Li <li.zekun@gmail.com> Co-authored-by: Aaron <lightmark@gmail.com> Co-authored-by: Brian R. Murphy <132495859+brmataptos@users.noreply.github.com> Co-authored-by: Oliver He <heliuchuan@gmail.com> Co-authored-by: David Wolinsky <isaac.wolinsky@gmail.com> Co-authored-by: Rati Gelashvili <gelash@users.noreply.github.com> Co-authored-by: runtianz <runtian@aptoslabs.com> Co-authored-by: Jin <128556004+0xjinn@users.noreply.github.com> Co-authored-by: Balaji Arun <balajia@vt.edu> Co-authored-by: Gerardo Di Giacomo <gerardo@aptoslabs.com> Co-authored-by: gedigi <gedigi@users.noreply.github.com> Co-authored-by: Greg Nazario <greg@gnazar.io> Co-authored-by: Michael Straka <mstraka100@gmail.com> Co-authored-by: Teng Zhang <rahxephon89@163.com> Co-authored-by: Josh Lind <josh.lind@hotmail.com> Co-authored-by: Brian (Sunghoon) Cho <brian@aptoslabs.com> Co-authored-by: Rustie Lin <rustie117@gmail.com> Co-authored-by: Guoteng Rao <3603304+grao1991@users.noreply.github.com> Co-authored-by: Perry Randall <perryjrandall@gmail.com> Co-authored-by: igor-aptos <110557261+igor-aptos@users.noreply.github.com> Co-authored-by: aalok-t <140445856+aalok-t@users.noreply.github.com> Co-authored-by: Aalok Thakkar <aalok@Aaloks-MacBook-Pro.local> Co-authored-by: Matt <90358481+xbtmatt@users.noreply.github.com> Co-authored-by: Wolfgang Grieskamp <wg@aptoslabs.com> Co-authored-by: Christian Theilemann <christian@aptoslabs.com> Co-authored-by: Victor Gao <10379359+vgao1996@users.noreply.github.com> Co-authored-by: larry-aptos <112209412+larry-aptos@users.noreply.github.com> Co-authored-by: Sital Kedia <sitalkedia@users.noreply.github.com> Co-authored-by: Manu Dhundi <manudhundi@gmail.com> Co-authored-by: 0xbe1 <0xbetrue@gmail.com> Co-authored-by: Kevin <105028215+movekevin@users.noreply.github.com>

aptos-labs#10274) [Sharded-Execution-GRPC] Add GRPC communication for sharded execution In this commit we replace the existing socket based communication (that is message send and message recv) with GRPC. Here we get the basic GRPC reliability. More reliability and better performance to come in subsequent commits

manudhundi force-pushed the manu/grpc_remote_executor branch 3 times, most recently from 32ca8ee to c48e1fb Compare September 27, 2023 20:44

manudhundi requested review from sitalkedia and zjma September 27, 2023 20:44

manudhundi self-assigned this Sep 27, 2023

manudhundi marked this pull request as ready for review September 27, 2023 20:46

manudhundi requested review from msmouse and lightmark as code owners September 27, 2023 20:46

sitalkedia reviewed Sep 27, 2023

View reviewed changes

grao1991 reviewed Sep 27, 2023

View reviewed changes

crates/aptos-protos/proto/aptos/remote_executor/v1/network_msg.proto Show resolved Hide resolved

crates/aptos-protos/src/pb/aptos.remote_executor.v1.rs Show resolved Hide resolved

crates/aptos-protos/proto/aptos/remote_executor/v1/network_msg.proto Outdated Show resolved Hide resolved

manudhundi force-pushed the manu/grpc_remote_executor branch from c085d88 to c962909 Compare September 29, 2023 07:00

sitalkedia reviewed Sep 29, 2023

View reviewed changes

sitalkedia approved these changes Oct 2, 2023

View reviewed changes

msmouse approved these changes Oct 3, 2023

View reviewed changes

manudhundi added 3 commits October 3, 2023 13:47

[Sharded-Execution-GRPC] Add GRPC communication for sharded execution

14e79f7

In this commit we replace the existing socket based communication with GRPC. Here we get the basic GRPC reliability. More reliability and better performance to come in subsequent commits

[Sharded-Execution-GRPC] Addressing comments of previous commit

84e0088

[Sharded-Execution-GRPC] Address some review comments

610363e

manudhundi force-pushed the manu/grpc_remote_executor branch from c453975 to 610363e Compare October 3, 2023 20:51

manudhundi enabled auto-merge (squash) October 3, 2023 21:34

This comment has been minimized.

Sign in to view

manudhundi merged commit 87496d2 into main Oct 3, 2023
85 checks passed

manudhundi deleted the manu/grpc_remote_executor branch October 3, 2023 22:05


		retry(fixed_retry_strategy(5, 20), \|\| network_client.write(&msg)).unwrap();
		if message_type.get_type() == "stop_task" {

[Sharded-Execution-GRPC] Add GRPC communication for sharded execution. #10274

[Sharded-Execution-GRPC] Add GRPC communication for sharded execution. #10274

Conversation

manudhundi commented Sep 27, 2023

Description

Test Plan

sitalkedia left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

manudhundi Sep 29, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

manudhundi Sep 29, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sitalkedia left a comment

Choose a reason for hiding this comment

manudhundi commented Oct 2, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Oct 3, 2023

✅ Forge suite compat success on aptos-node-v1.6.2 ==> 610363e5a86578196cdfd6519d9ffb06a5bf1cb8

github-actions bot commented Oct 3, 2023

✅ Forge suite realistic_env_max_load success on 610363e5a86578196cdfd6519d9ffb06a5bf1cb8

github-actions bot commented Oct 3, 2023

✅ Forge suite framework_upgrade success on aptos-node-v1.5.1 ==> 610363e5a86578196cdfd6519d9ffb06a5bf1cb8

manudhundi Sep 29, 2023 •

edited

Loading

manudhundi Sep 29, 2023 •

edited

Loading

✅ Forge suite `compat` success on `aptos-node-v1.6.2` ==> `610363e5a86578196cdfd6519d9ffb06a5bf1cb8`

✅ Forge suite `realistic_env_max_load` success on `610363e5a86578196cdfd6519d9ffb06a5bf1cb8`

✅ Forge suite `framework_upgrade` success on `aptos-node-v1.5.1` ==> `610363e5a86578196cdfd6519d9ffb06a5bf1cb8`