Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

markuze/aptos perf #5998

Closed
wants to merge 23 commits into from
Closed

markuze/aptos perf #5998

wants to merge 23 commits into from

Conversation

Markuze
Copy link
Contributor

@Markuze Markuze commented Dec 26, 2022

Description

Test Plan

@Markuze Markuze force-pushed the markuze/aptos-perf branch 2 times, most recently from eaafd02 to 2ea39d3 Compare December 26, 2022 20:04
@Markuze Markuze added the CICD:run-e2e-tests when this label is present github actions will run all land-blocking e2e tests from the PR label Dec 27, 2022
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@Markuze Markuze force-pushed the markuze/aptos-perf branch 2 times, most recently from 57537ce to b33d4b6 Compare December 28, 2022 20:50
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@Markuze Markuze force-pushed the markuze/aptos-perf branch 2 times, most recently from 8ff9c9e to 3612a30 Compare January 3, 2023 17:19
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions
Copy link
Contributor

github-actions bot commented Jan 3, 2023

❌ Forge suite compat failure on testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 39777f6448640b91dd925374d41532025f25a23d

Compatibility test results for testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 39777f6448640b91dd925374d41532025f25a23d (PR)
1. Check liveness of validators at old version: testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b
compatibility::simple-validator-upgrade::liveness-check : 7008 TPS, 5533 ms latency, 8700 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 39777f6448640b91dd925374d41532025f25a23d
Test Failed: Timed out waiting for Node validator-1:b502e2300dfa1947fd30ef869d235e59398e32e84b68e004ab7982a880fbb477 to be healthy
Trailing Log Lines:
::error::Timed out waiting for Node validator-1:b502e2300dfa1947fd30ef869d235e59398e32e84b68e004ab7982a880fbb477 to be healthy
test compatibility::simple-validator-upgrade ... FAILED
Error: Timed out waiting for Node validator-1:b502e2300dfa1947fd30ef869d235e59398e32e84b68e004ab7982a880fbb477 to be healthy
Test Statistics: 
Compatibility test results for testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 39777f6448640b91dd925374d41532025f25a23d (PR)
1. Check liveness of validators at old version: testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b
compatibility::simple-validator-upgrade::liveness-check : 7008 TPS, 5533 ms latency, 8700 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 39777f6448640b91dd925374d41532025f25a23d
Test Failed: Timed out waiting for Node validator-1:b502e2300dfa1947fd30ef869d235e59398e32e84b68e004ab7982a880fbb477 to be healthy


Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:280"},"thread_name":"main","hostname":"forge-compat-pr-5998-1672777728-testnet-2d8b1b57553d869190f61df","timestamp":"2023-01-03T20:34:35.437609Z","message":"Deleting namespace forge-compat-pr-5998: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:388"},"thread_name":"main","hostname":"forge-compat-pr-5998-1672777728-testnet-2d8b1b57553d869190f61df","timestamp":"2023-01-03T20:34:35.437633Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-5998"}

failures:
    compatibility::simple-validator-upgrade

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:

@github-actions
Copy link
Contributor

github-actions bot commented Jan 3, 2023

❌ Forge suite land_blocking failure on 39777f6448640b91dd925374d41532025f25a23d

Forge test runner terminated:
Trailing Log Lines:
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/stateful_set.rs:157"},"thread_name":"main","hostname":"forge-e2e-pr-5998-1672777662-39777f6448640b91dd925374d41532025f","timestamp":"2023-01-03T20:33:51.581337Z","message":"Pod aptos-node-18-validator-0 at phase Running"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/stateful_set.rs:107"},"thread_name":"main","hostname":"forge-e2e-pr-5998-1672777662-39777f6448640b91dd925374d41532025f","timestamp":"2023-01-03T20:34:51.586752Z","message":"StatefulSet aptos-node-18-validator has 0/1 replicas"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/stateful_set.rs:157"},"thread_name":"main","hostname":"forge-e2e-pr-5998-1672777662-39777f6448640b91dd925374d41532025f","timestamp":"2023-01-03T20:34:51.589444Z","message":"Pod aptos-node-18-validator-0 at phase Running"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/stateful_set.rs:107"},"thread_name":"main","hostname":"forge-e2e-pr-5998-1672777662-39777f6448640b91dd925374d41532025f","timestamp":"2023-01-03T20:35:51.596279Z","message":"StatefulSet aptos-node-18-validator has 0/1 replicas"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/stateful_set.rs:133"},"thread_name":"main","hostname":"forge-e2e-pr-5998-1672777662-39777f6448640b91dd925374d41532025f","timestamp":"2023-01-03T20:35:51.599068Z","message":"Pod aptos-node-18-validator-0 has CrashLoopBackOff"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:280"},"thread_name":"main","hostname":"forge-e2e-pr-5998-1672777662-39777f6448640b91dd925374d41532025f","timestamp":"2023-01-03T20:35:51.618841Z","message":"Deleting namespace forge-e2e-pr-5998: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:388"},"thread_name":"main","hostname":"forge-e2e-pr-5998-1672777662-39777f6448640b91dd925374d41532025f","timestamp":"2023-01-03T20:35:51.618866Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-5998"}
Failed to run tests:
CrashLoopBackOff
Error: CrashLoopBackOff
Debugging output:
NAME                                    READY   STATUS             RESTARTS      AGE
aptos-node-0-fullnode-eforge192-0       1/1     Running            0             4m21s
aptos-node-0-validator-0                0/1     CrashLoopBackOff   4 (82s ago)   4m21s
aptos-node-1-fullnode-eforge192-0       1/1     Running            0             4m21s
aptos-node-1-validator-0                0/1     CrashLoopBackOff   4 (84s ago)   4m21s
aptos-node-10-validator-0               0/1     CrashLoopBackOff   4 (84s ago)   4m21s
aptos-node-11-validator-0               0/1     CrashLoopBackOff   4 (78s ago)   4m20s
aptos-node-12-validator-0               0/1     CrashLoopBackOff   4 (75s ago)   4m21s
aptos-node-13-validator-0               0/1     CrashLoopBackOff   4 (89s ago)   4m20s
aptos-node-14-validator-0               0/1     CrashLoopBackOff   4 (87s ago)   4m21s
aptos-node-15-validator-0               0/1     CrashLoopBackOff   4 (78s ago)   4m21s
aptos-node-16-validator-0               0/1     CrashLoopBackOff   4 (79s ago)   4m20s
aptos-node-17-validator-0               0/1     CrashLoopBackOff   4 (85s ago)   4m21s
aptos-node-18-validator-0               0/1     CrashLoopBackOff   4 (78s ago)   4m21s
aptos-node-19-validator-0               0/1     CrashLoopBackOff   4 (91s ago)   4m21s
aptos-node-2-fullnode-eforge192-0       1/1     Running            0             4m20s
aptos-node-2-validator-0                0/1     CrashLoopBackOff   4 (90s ago)   4m20s
aptos-node-3-fullnode-eforge192-0       1/1     Running            0             4m20s
aptos-node-3-validator-0                0/1     Error              5 (89s ago)   4m21s
aptos-node-4-fullnode-eforge192-0       1/1     Running            0             4m20s
aptos-node-4-validator-0                0/1     CrashLoopBackOff   4 (76s ago)   4m21s
aptos-node-5-fullnode-eforge192-0       1/1     Running            0             4m21s
aptos-node-5-validator-0                0/1     CrashLoopBackOff   4 (83s ago)   4m21s
aptos-node-6-fullnode-eforge192-0       1/1     Running            0             4m21s
aptos-node-6-validator-0                0/1     Error              5 (92s ago)   4m21s
aptos-node-7-fullnode-eforge192-0       1/1     Running            0             4m21s
aptos-node-7-validator-0                0/1     CrashLoopBackOff   4 (77s ago)   4m20s
aptos-node-8-fullnode-eforge192-0       1/1     Running            0             4m21s
aptos-node-8-validator-0                0/1     Error              5 (92s ago)   4m21s
aptos-node-9-fullnode-eforge192-0       1/1     Running            0             4m20s
aptos-node-9-validator-0                0/1     Error              5 (85s ago)   4m20s
genesis-aptos-genesis-eforge192-4s8ql   0/1     Completed          0             7m16s

@Markuze Markuze closed this Jan 4, 2023
@Markuze
Copy link
Contributor Author

Markuze commented Jan 4, 2023

Dev branch. Moved to markuze/aptos_perf_pr
#6054

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CICD:run-e2e-tests when this label is present github actions will run all land-blocking e2e tests from the PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant