Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

dial tcp 172.16.0.13:9000: i/o timeout #694

Open
davidLeeeeeeeeeeee opened this issue Jun 27, 2024 · 9 comments
Open

dial tcp 172.16.0.13:9000: i/o timeout #694

davidLeeeeeeeeeeee opened this issue Jun 27, 2024 · 9 comments

Comments

@davidLeeeeeeeeeeee
Copy link

Ubuntu 22.04 LTS

Docker version 20.10.21, build 20.10.21-0ubuntu1~22.04.3

kurtosis-cli version (0.90.1)

(base) root@ST-ZY-28:/var/zk_evm# kurtosis run github.com/ethpandaops/ethereum-package --args-file ./network_params2.yaml --image-download always
INFO[2024-06-27T07:36:57+08:00] Creating a new enclave for Starlark to run inside... 
INFO[2024-06-27T07:37:03+08:00] Enclave 'dry-geyser' created successfully    

Container images used in this run:
> ethereum/client-go:latest - remotely downloaded
> ethpandaops/ethereum-genesis-generator:3.3.5 - remotely downloaded
> python:3.11-alpine - remotely downloaded
> protolambda/eth2-val-tools:latest - remotely downloaded
> badouralix/curl-jq - remotely downloaded
> consensys/teku:latest - remotely downloaded
> ethpandaops/dora:latest - remotely downloaded
> ethpandaops/lighthouse:stable - remotely downloaded

Uploading file '/static_files/jwt/jwtsecret' to files artifact 'jwt_file'
Files with artifact name 'jwt_file' uploaded with artifact UUID 'bda9564aa9874d3eb00d151b57365786'

Uploading file '/static_files/keymanager/keymanager.txt' to files artifact 'keymanager_file'
Files with artifact name 'keymanager_file' uploaded with artifact UUID 'e7799dd9ca6847f9b1a4e007738d1209'

Printing a message
Read the prometheus, grafana templates

Printing a message
Launching participant network with 3 participants and the following network params struct(additional_preloaded_contracts = {}, churn_limit_quotient = 65536, custody_requirement = 1, data_column_sidecar_subnet_count = 32, deneb_fork_epoch = 0, deposit_contract_address = "0x4242424242424242424242424242424242424242", devnet_repo = "ethpandaops", eip7594_fork_epoch = 100000001, eip7594_fork_version = "0x70000038", ejection_balance = 16000000000, electra_fork_epoch = 100000000, eof_activation_epoch = "", eth1_follow_distance = 2048, genesis_delay = 20, max_per_epoch_activation_churn_limit = 8, min_validator_withdrawability_delay = 256, network = "kurtosis", network_id = "585858", network_sync_base_url = "https://ethpandaops-ethereum-node-snapshots.ams3.cdn.digitaloceanspaces.com/", num_validator_keys_per_node = 64, preregistered_validator_count = 0, preregistered_validator_keys_mnemonic = "giant issue aisle success illegal bike spike question tent bar rely arctic volcano long crawl hungry vocal artwork sniff fantasy very lucky have athlete", preset = "mainnet", samples_per_slot = 8, seconds_per_slot = 12, shard_committee_period = 256, target_number_of_peers = 70)

Printing a message
Generating cl validator key stores

Adding service with name 'validator-key-generation-cl-validator-keystore' and image 'protolambda/eth2-val-tools:latest'
Service 'validator-key-generation-cl-validator-keystore' added with service UUID '154ff54a408d40af813a95b837989070'

Generating keystores
Command returned with exit code '0' with no output

Verifying whether two values meet a certain condition '=='
Verification succeeded. Value is '0'.

Storing files from service 'validator-key-generation-cl-validator-keystore' at path '/node-0-keystores/' to files artifact with name '1-lighthouse-geth-0-63-0'
Files with artifact name '1-lighthouse-geth-0-63-0' uploaded with artifact UUID 'bb406afedb3a433da318e2d887bc74cd'

Storing files from service 'validator-key-generation-cl-validator-keystore' at path '/node-1-keystores/' to files artifact with name '2-lighthouse-geth-64-127-0'
Files with artifact name '2-lighthouse-geth-64-127-0' uploaded with artifact UUID '0c1bef28eb784981972f6a411fdca353'

Storing files from service 'validator-key-generation-cl-validator-keystore' at path '/node-2-keystores/' to files artifact with name '3-teku-geth-128-191-0'
Files with artifact name '3-teku-geth-128-191-0' uploaded with artifact UUID 'a86050e797794aeea32fe31d6e415ab7'

Storing prysm password in a file
Command returned with exit code '0' with no output

Verifying whether two values meet a certain condition '=='
Verification succeeded. Value is '0'.

Storing files from service 'validator-key-generation-cl-validator-keystore' at path '/tmp/prysm-password.txt' to files artifact with name 'prysm-password'
Files with artifact name 'prysm-password' uploaded with artifact UUID '36c3d12fe1aa467da10e41f4c1ca1a99'

Printing a message
{
        "per_node_keystores": [
                {
                        "files_artifact_uuid": "1-lighthouse-geth-0-63-0",
                        "nimbus_keys_relative_dirpath": "/nimbus-keys",
                        "prysm_relative_dirpath": "/prysm",
                        "raw_keys_relative_dirpath": "/keys",
                        "raw_root_dirpath": "",
                        "raw_secrets_relative_dirpath": "/secrets",
                        "teku_keys_relative_dirpath": "/teku-keys",
                        "teku_secrets_relative_dirpath": "/teku-secrets"
                },
                {
                        "files_artifact_uuid": "2-lighthouse-geth-64-127-0",
                        "nimbus_keys_relative_dirpath": "/nimbus-keys",
                        "prysm_relative_dirpath": "/prysm",
                        "raw_keys_relative_dirpath": "/keys",
                        "raw_root_dirpath": "",
                        "raw_secrets_relative_dirpath": "/secrets",
                        "teku_keys_relative_dirpath": "/teku-keys",
                        "teku_secrets_relative_dirpath": "/teku-secrets"
                },
                {
                        "files_artifact_uuid": "3-teku-geth-128-191-0",
                        "nimbus_keys_relative_dirpath": "/nimbus-keys",
                        "prysm_relative_dirpath": "/prysm",
                        "raw_keys_relative_dirpath": "/keys",
                        "raw_root_dirpath": "",
                        "raw_secrets_relative_dirpath": "/secrets",
                        "teku_keys_relative_dirpath": "/teku-keys",
                        "teku_secrets_relative_dirpath": "/teku-secrets"
                }
        ],
        "prysm_password_artifact_uuid": "prysm-password",
        "prysm_password_relative_filepath": "prysm-password.txt"
}

Getting final genesis timestamp
Command returned with exit code '0' and the following output: 1719445084

Printing a message
Generating EL CL data

Rendering a template to a files artifact with name 'genesis-el-cl-env-file'
Templates artifact name 'genesis-el-cl-env-file' rendered with artifact UUID '6e53c4476eb241de9e230965012265f1'

Creating genesis
Command returned with exit code '0' and the following output:
--------------------
+ '[' -f /data/metadata/genesis.json ']'
++ mktemp -d -t ci-XXXXXXXXXX
+ tmp_dir=/tmp/ci-JVD5dzzsSr
+ mkdir -p /data/metadata
+ envsubst
+ python3 /apps/el-gen/genesis_geth.py /tmp/ci-JVD5dzzsSr/genesis-config.yaml
+ python3 /apps/el-gen/genesis_chainspec.py /tmp/ci-JVD5dzzsSr/genesis-config.yaml
+ python3 /apps/el-gen/genesis_besu.py /tmp/ci-JVD5dzzsSr/genesis-config.yaml
+ gen_cl_config
+ . /apps/el-gen/.venv/bin/activate
++ deactivate nondestructive
++ '[' -n /root/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ']'
++ PATH=/root/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ export PATH
++ unset _OLD_VIRTUAL_PATH
++ '[' -n '' ']'
++ '[' -n /bin/bash -o -n '' ']'
++ hash -r
++ '[' -n '' ']'
++ unset VIRTUAL_ENV
++ unset VIRTUAL_ENV_PROMPT
++ '[' '!' nondestructive = nondestructive ']'
++ VIRTUAL_ENV=/apps/el-gen/.venv
++ export VIRTUAL_ENV
++ _OLD_VIRTUAL_PATH=/root/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ PATH=/apps/el-gen/.venv/bin:/root/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ export PATH
++ '[' -n '' ']'
++ '[' -z '' ']'
++ _OLD_VIRTUAL_PS1='(.venv) '
++ PS1='(.venv) (.venv) '
++ export PS1
++ VIRTUAL_ENV_PROMPT='(.venv) '
++ export VIRTUAL_ENV_PROMPT
++ '[' -n /bin/bash -o -n '' ']'
++ hash -r
+ set -x
+ '[' -f /data/metadata/genesis.ssz ']'
++ mktemp -d -t ci-XXXXXXXXXX
+ tmp_dir=/tmp/ci-D9DSveuswY
+ mkdir -p /data/metadata
+ mkdir -p /data/parsed
+ envsubst
+ envsubst
+ [[ mainnet == \m\i\n\i\m\a\l ]]
+ cp /tmp/ci-D9DSveuswY/mnemonics.yaml /data/metadata/mnemonics.yaml
+ grep DEPOSIT_CONTRACT_ADDRESS /data/metadata/config.yaml
+ cut -d ' ' -f2
+ echo 0
+ echo enr:-Iq4QJk4WqRkjsX5c2CXtOra6HnxN-BMXnWhmhEQO9Bn9iABTJGdjUOurM7Btj1ouKaFkvTRoju5vz2GPmVON2dffQKGAX53x8JigmlkgnY0gmlwhLKAlv6Jc2VjcDI1NmsxoQK6S-Cii_KmfFdUJL2TANL3ksaKUnNXvTCv1tLwXs0QgIN1ZHCCIyk
+ envsubst
+ genesis_args=(deneb --config /data/metadata/config.yaml --mnemonics $tmp_dir/mnemonics.yaml --tranches-dir /data/metadata/tranches --state-output /data/metadata/genesis.ssz --preset-phase0 $PRESET_BASE --preset-altair $PRESET_BASE --preset-bellatrix $PRESET_BASE --preset-capella $PRESET_BASE --preset-deneb $PRESET_BASE)
+ [[ 0x00 == \0\x\0\1 ]]
+ [[ '' != '' ]]
+ [[ '' != '' ]]
+ genesis_args+=(--eth1-config /data/metadata/genesis.json)
+ '[' -z '' ']'
+ zcli_args=(pretty deneb BeaconState --preset-phase0 $PRESET_BASE --preset-altair $PRESET_BASE --preset-bellatrix $PRESET_BASE --preset-capella $PRESET_BASE --preset-deneb $PRESET_BASE /data/metadata/genesis.ssz)
+ /usr/local/bin/eth2-testnet-genesis deneb --config /data/metadata/config.yaml --mnemonics /tmp/ci-D9DSveuswY/mnemonics.yaml --tranches-dir /data/metadata/tranches --state-output /data/metadata/genesis.ssz --preset-phase0 mainnet --preset-altair mainnet --preset-bellatrix mainnet --preset-capella mainnet --preset-deneb mainnet --eth1-config /data/metadata/genesis.json
zrnt version: v0.32.3
Using CL MIN_GENESIS_TIME for genesis timestamp
processing mnemonic 0, for 192 validators
...validator 100/192
Writing pubkeys list file...
generated 192 validators from mnemonic yaml (/tmp/ci-D9DSveuswY/mnemonics.yaml)
eth2 genesis at 1719445084 + 20 = 1719445104  (2024-06-26 23:38:24 +0000 UTC)
done preparing state, serializing SSZ now...
done!
+ /usr/local/bin/zcli pretty deneb BeaconState --preset-phase0 mainnet --preset-altair mainnet --preset-bellatrix mainnet --preset-capella mainnet --preset-deneb mainnet /data/metadata/genesis.ssz
+ echo 'Genesis args: deneb' --config /data/metadata/config.yaml --mnemonics /tmp/ci-D9DSveuswY/mnemonics.yaml --tranches-dir /data/metadata/tranches --state-output /data/metadata/genesis.ssz --preset-phase0 mainnet --preset-altair mainnet --preset-bellatrix mainnet --preset-capella mainnet --preset-deneb mainnet --eth1-config /data/metadata/genesis.json
Genesis args: deneb --config /data/metadata/config.yaml --mnemonics /tmp/ci-D9DSveuswY/mnemonics.yaml --tranches-dir /data/metadata/tranches --state-output /data/metadata/genesis.ssz --preset-phase0 mainnet --preset-altair mainnet --preset-bellatrix mainnet --preset-capella mainnet --preset-deneb mainnet --eth1-config /data/metadata/genesis.json
++ jq -r .latest_execution_payload_header.block_number /data/parsed/parsedConsensusGenesis.json
Genesis block number: 0
+ echo 'Genesis block number: 0'
++ jq -r .latest_execution_payload_header.block_hash /data/parsed/parsedConsensusGenesis.json
Genesis block hash: 0x84a5b804f1d7ff56117d18e47ea2d8709b2524fdfc797f1701295d7cf634e1b1
+ echo 'Genesis block hash: 0x84a5b804f1d7ff56117d18e47ea2d8709b2524fdfc797f1701295d7cf634e1b1'
+ jq -r .eth1_data.block_hash /data/parsed/parsedConsensusGenesis.json
+ tr -d '\n'
+ jq -r .genesis_validators_root /data/parsed/parsedConsensusGenesis.json
+ tr -d '\n'
+ gen_shared_files
+ . /apps/el-gen/.venv/bin/activate
++ deactivate nondestructive
++ '[' -n /root/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ']'
++ PATH=/root/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ export PATH
++ unset _OLD_VIRTUAL_PATH
++ '[' -n '' ']'
++ '[' -n /bin/bash -o -n '' ']'
++ hash -r
++ '[' -n '(.venv) ' ']'
++ PS1='(.venv) '
++ export PS1
++ unset _OLD_VIRTUAL_PS1
++ unset VIRTUAL_ENV
++ unset VIRTUAL_ENV_PROMPT
++ '[' '!' nondestructive = nondestructive ']'
++ VIRTUAL_ENV=/apps/el-gen/.venv
++ export VIRTUAL_ENV
++ _OLD_VIRTUAL_PATH=/root/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ PATH=/apps/el-gen/.venv/bin:/root/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ export PATH
++ '[' -n '' ']'
++ '[' -z '' ']'
++ _OLD_VIRTUAL_PS1='(.venv) '
++ PS1='(.venv) (.venv) '
++ export PS1
++ VIRTUAL_ENV_PROMPT='(.venv) '
++ export VIRTUAL_ENV_PROMPT
++ '[' -n /bin/bash -o -n '' ']'
++ hash -r
+ set -x
+ mkdir -p /data/metadata
+ '[' -f /data/jwt/jwtsecret ']'
+ mkdir -p /data/jwt
++ openssl rand -hex 32
++ tr -d '\n'
+ echo -n 0xaab62aed5f4914d74de359ce8c327f2ead6685ffbbc062ea793f0a3e7ce93cff
+ '[' -f /data/metadata/genesis.json ']'
++ cat /data/metadata/genesis.json
++ jq -r '.config.terminalTotalDifficulty | tostring'
+ terminalTotalDifficulty=0
+ sed -i 's/TERMINAL_TOTAL_DIFFICULTY:.*/TERMINAL_TOTAL_DIFFICULTY: 0/' /data/metadata/config.yaml
+ '[' false = true ']'

--------------------

Reading genesis validators root
Command returned with exit code '0' and the following output: 0xa63a1b2ffefb0fea452e9ad530c9a4bbd58c97c6ad82da001f142be64c91d61b

Reading prague time from genesis
Command returned with exit code '0' and the following output: 40119445104

Adding service with name 'el-1-geth-lighthouse' and image 'ethereum/client-go:latest'
Service 'el-1-geth-lighthouse' added with service UUID '294aacfef50642ff9b63bd3f9e45309c'

Waiting for at most '15m0s' for service 'el-1-geth-lighthouse' to reach a certain state
Wait took 1 tries (38.156124ms in total). Assertion passed with following:
Request had response code '200' and body "{\"jsonrpc\":\"2.0\",\"id\":1,\"result\":{\"id\":\"0015779b9c938aa4c39453251c0987901cdebd498506297aeb696fc34c2a7459\",\"name\":\"Geth/v1.14.6-unstable-9298d2db/linux-amd64/go1.22.4\",\"enode\":\"enode://68f23d016ec17b6cdaaff13c9b867051289ea51846a8dd132a74c9d4720a18adcfbc33ebbb9647a25812104aeee6013ba014f0581ecd41c5b913281bf628eb68@172.16.0.10:30303\",\"enr\":\"enr:-Ki4QCZ_HN4PY05VtnQhg2dRk_7JA7TIvHAOej_vh0J3naakPUMvHJhxPakQtr9QI9n64INgX1OdOvuJQ3xHO9ttYBiGAZBW6dmHg2V0aMzLhCNZ8SqFCVdOJnCCaWSCdjSCaXCErBAAColzZWNwMjU2azGhAmjyPQFuwXts2q_xPJuGcFEonqUYRqjdEyp0ydRyChithHNuYXDAg3RjcIJ2X4N1ZHCCdl8\",\"ip\":\"172.16.0.10\",\"ports\":{\"discovery\":30303,\"listener\":30303},\"listenAddr\":\"[::]:30303\",\"protocols\":{\"eth\":{\"network\":585858,\"difficulty\":1,\"genesis\":\"0x84a5b804f1d7ff56117d18e47ea2d8709b2524fdfc797f1701295d7cf634e1b1\",\"config\":{\"chainId\":585858,\"homesteadBlock\":0,\"eip150Block\":0,\"eip155Block\":0,\"eip158Block\":0,\"byzantiumBlock\":0,\"constantinopleBlock\":0,\"petersburgBlock\":0,\"istanbulBlock\":0,\"berlinBlock\":0,\"londonBlock\":0,\"mergeNetsplitBlock\":0,\"shanghaiTime\":0,\"cancunTime\":0,\"pragueTime\":40119445104,\"terminalTotalDifficulty\":0,\"terminalTotalDifficultyPassed\":true},\"head\":\"0x84a5b804f1d7ff56117d18e47ea2d8709b2524fdfc797f1701295d7cf634e1b1\"},\"snap\":{}}}}\n", with extracted fields:
'extract.enr': "enr:-Ki4QCZ_HN4PY05VtnQhg2dRk_7JA7TIvHAOej_vh0J3naakPUMvHJhxPakQtr9QI9n64INgX1OdOvuJQ3xHO9ttYBiGAZBW6dmHg2V0aMzLhCNZ8SqFCVdOJnCCaWSCdjSCaXCErBAAColzZWNwMjU2azGhAmjyPQFuwXts2q_xPJuGcFEonqUYRqjdEyp0ydRyChithHNuYXDAg3RjcIJ2X4N1ZHCCdl8"
'extract.enode': "enode://68f23d016ec17b6cdaaff13c9b867051289ea51846a8dd132a74c9d4720a18adcfbc33ebbb9647a25812104aeee6013ba014f0581ecd41c5b913281bf628eb68@172.16.0.10:30303"

Adding service with name 'el-2-geth-lighthouse' and image 'ethereum/client-go:latest'
Service 'el-2-geth-lighthouse' added with service UUID '21e3516db10e4e69ae126cae31086e24'

Waiting for at most '15m0s' for service 'el-2-geth-lighthouse' to reach a certain state
Wait took 1 tries (54.889823ms in total). Assertion passed with following:
Request had response code '200' and body "{\"jsonrpc\":\"2.0\",\"id\":1,\"result\":{\"id\":\"c0cee76f0fea4deb74e3def28ec53eab34447f09bc18914a270b0b589daf2b02\",\"name\":\"Geth/v1.14.6-unstable-9298d2db/linux-amd64/go1.22.4\",\"enode\":\"enode://4b59e59dcac69c2ab50155e9a14e6caaf72f8da38685f666ec12f8ba21c2a737cca41295a15034c05be1acd1883b1dc255050563d4b1a46d676d3c6427a752b0@172.16.0.11:30303\",\"enr\":\"enr:-Ki4QNzXjqxeBjdql6Lpwo2PJbY4ASRBH3R_mPwJ57_pDt4rGLFaBeQB5BNp9LKUtxc0g9bYJnOYkb5UnD3TvxczYDiGAZBW6fD8g2V0aMzLhCNZ8SqFCVdOJnCCaWSCdjSCaXCErBAAC4lzZWNwMjU2azGhAktZ5Z3KxpwqtQFV6aFObKr3L42jhoX2ZuwS-Lohwqc3hHNuYXDAg3RjcIJ2X4N1ZHCCdl8\",\"ip\":\"172.16.0.11\",\"ports\":{\"discovery\":30303,\"listener\":30303},\"listenAddr\":\"[::]:30303\",\"protocols\":{\"eth\":{\"network\":585858,\"difficulty\":1,\"genesis\":\"0x84a5b804f1d7ff56117d18e47ea2d8709b2524fdfc797f1701295d7cf634e1b1\",\"config\":{\"chainId\":585858,\"homesteadBlock\":0,\"eip150Block\":0,\"eip155Block\":0,\"eip158Block\":0,\"byzantiumBlock\":0,\"constantinopleBlock\":0,\"petersburgBlock\":0,\"istanbulBlock\":0,\"berlinBlock\":0,\"londonBlock\":0,\"mergeNetsplitBlock\":0,\"shanghaiTime\":0,\"cancunTime\":0,\"pragueTime\":40119445104,\"terminalTotalDifficulty\":0,\"terminalTotalDifficultyPassed\":true},\"head\":\"0x84a5b804f1d7ff56117d18e47ea2d8709b2524fdfc797f1701295d7cf634e1b1\"},\"snap\":{}}}}\n", with extracted fields:
'extract.enr': "enr:-Ki4QNzXjqxeBjdql6Lpwo2PJbY4ASRBH3R_mPwJ57_pDt4rGLFaBeQB5BNp9LKUtxc0g9bYJnOYkb5UnD3TvxczYDiGAZBW6fD8g2V0aMzLhCNZ8SqFCVdOJnCCaWSCdjSCaXCErBAAC4lzZWNwMjU2azGhAktZ5Z3KxpwqtQFV6aFObKr3L42jhoX2ZuwS-Lohwqc3hHNuYXDAg3RjcIJ2X4N1ZHCCdl8"
'extract.enode': "enode://4b59e59dcac69c2ab50155e9a14e6caaf72f8da38685f666ec12f8ba21c2a737cca41295a15034c05be1acd1883b1dc255050563d4b1a46d676d3c6427a752b0@172.16.0.11:30303"

Adding service with name 'el-3-geth-teku' and image 'ethereum/client-go:latest'
Service 'el-3-geth-teku' added with service UUID '1392da84cc1e49889f462480cf5a3e4b'

Waiting for at most '15m0s' for service 'el-3-geth-teku' to reach a certain state
Wait took 1 tries (46.107772ms in total). Assertion passed with following:
Request had response code '200' and body "{\"jsonrpc\":\"2.0\",\"id\":1,\"result\":{\"id\":\"b5b793c6865d58156f3d1019c50897f5a9dd12946f4a69cb4eb928ed0b69c73b\",\"name\":\"Geth/v1.14.6-unstable-9298d2db/linux-amd64/go1.22.4\",\"enode\":\"enode://bf7dff3e8fc30d7496cb084871203135063ffe17c3b5b467755428577dadf7a0b17cbc157e9e1edbf0dadcdc6c4e2a6a73cec2e2a34774379d7fc77b8b089462@172.16.0.12:30303\",\"enr\":\"enr:-Ki4QOgwbxm8dvivQ0HhHq3sVqt5BFwGw8R4lNMSrvoGzlKhLzPe82D_UegFkJEMIeH8GZmdSqzrrJxlT73NxdXjnSOGAZBW6gjgg2V0aMzLhCNZ8SqFCVdOJnCCaWSCdjSCaXCErBAADIlzZWNwMjU2azGhAr99_z6Pww10lssISHEgMTUGP_4Xw7W0Z3VUKFd9rfeghHNuYXDAg3RjcIJ2X4N1ZHCCdl8\",\"ip\":\"172.16.0.12\",\"ports\":{\"discovery\":30303,\"listener\":30303},\"listenAddr\":\"[::]:30303\",\"protocols\":{\"eth\":{\"network\":585858,\"difficulty\":1,\"genesis\":\"0x84a5b804f1d7ff56117d18e47ea2d8709b2524fdfc797f1701295d7cf634e1b1\",\"config\":{\"chainId\":585858,\"homesteadBlock\":0,\"eip150Block\":0,\"eip155Block\":0,\"eip158Block\":0,\"byzantiumBlock\":0,\"constantinopleBlock\":0,\"petersburgBlock\":0,\"istanbulBlock\":0,\"berlinBlock\":0,\"londonBlock\":0,\"mergeNetsplitBlock\":0,\"shanghaiTime\":0,\"cancunTime\":0,\"pragueTime\":40119445104,\"terminalTotalDifficulty\":0,\"terminalTotalDifficultyPassed\":true},\"head\":\"0x84a5b804f1d7ff56117d18e47ea2d8709b2524fdfc797f1701295d7cf634e1b1\"},\"snap\":{}}}}\n", with extracted fields:
'extract.enode': "enode://bf7dff3e8fc30d7496cb084871203135063ffe17c3b5b467755428577dadf7a0b17cbc157e9e1edbf0dadcdc6c4e2a6a73cec2e2a34774379d7fc77b8b089462@172.16.0.12:30303"
'extract.enr': "enr:-Ki4QOgwbxm8dvivQ0HhHq3sVqt5BFwGw8R4lNMSrvoGzlKhLzPe82D_UegFkJEMIeH8GZmdSqzrrJxlT73NxdXjnSOGAZBW6gjgg2V0aMzLhCNZ8SqFCVdOJnCCaWSCdjSCaXCErBAADIlzZWNwMjU2azGhAr99_z6Pww10lssISHEgMTUGP_4Xw7W0Z3VUKFd9rfeghHNuYXDAg3RjcIJ2X4N1ZHCCdl8"

Printing a message
Successfully added 3 EL participants

Printing a message
Launching CL network

Adding service with name 'cl-1-lighthouse-geth' and image 'ethpandaops/lighthouse:stable'
There was an error executing Starlark code 
An error occurred executing instruction (number 30) at github.com/ethpandaops/ethereum-package/src/cl/lighthouse/lighthouse_launcher.star[161:38]:
  add_service(name="cl-1-lighthouse-geth", config=ServiceConfig(image="ethpandaops/lighthouse:stable", ports={"http": PortSpec(number=4000, transport_protocol="TCP", application_protocol="http"), "metrics": PortSpec(number=5054, transport_protocol="TCP", application_protocol="http"), "tcp-discovery": PortSpec(number=9000, transport_protocol="TCP", application_protocol=""), "udp-discovery": PortSpec(number=9000, transport_protocol="UDP", application_protocol="")}, public_ports={}, files={"/jwt": "jwt_file", "/network-configs": "el_cl_genesis_data"}, cmd=["lighthouse", "beacon_node", "--debug-level=info", "--datadir=/data/lighthouse/beacon-data", "--disable-enr-auto-update", "--enr-address=KURTOSIS_IP_ADDR_PLACEHOLDER", "--enr-udp-port=9000", "--enr-tcp-port=9000", "--listen-address=0.0.0.0", "--port=9000", "--http", "--http-address=0.0.0.0", "--http-port=4000", "--http-allow-sync-stalled", "--slots-per-restore-point=32", "--disable-packet-filter", "--execution-endpoints=http://{{kurtosis:8db3f51e913e4359817782acfe6d80e8:ip_address.runtime_value}}:8551", "--jwt-secrets=/jwt/jwtsecret", "--suggested-fee-recipient=0x8943545177806ED17B9F23F0a21ee5948eCaa776", "--subscribe-all-subnets", "--metrics", "--metrics-address=0.0.0.0", "--metrics-allow-origin=*", "--metrics-port=5054", "--enable-private-discovery", "--testnet-dir=/network-configs"], env_vars={"RUST_BACKTRACE": "full"}, private_ip_address_placeholder="KURTOSIS_IP_ADDR_PLACEHOLDER", max_cpu=1000, min_cpu=50, max_memory=1024, min_memory=256, ready_conditions=ReadyCondition(recipe=GetHttpRequestRecipe(port_id="http", endpoint="/eth/v1/node/health"), field="code", assertion="IN", target_value=[200, 206], timeout="15m"), labels={"ethereum-package.client": "lighthouse", "ethereum-package.client-image": "ethpandaops-lighthouse_stable", "ethereum-package.client-type": "beacon", "ethereum-package.connected-client": "geth", "ethereum-package.sha256": ""}, tolerations=[], node_selectors={}))
  Caused by: Unexpected error occurred starting service 'cl-1-lighthouse-geth'
  Caused by: An error occurred waiting for all TCP and UDP ports to be open for service 'cl-1-lighthouse-geth' with private IP '172.16.0.13'; this is usually due to a misconfiguration in the service itself, so here are the logs:
  == SERVICE 'cl-1-lighthouse-geth' LOGS ===================================
  Jun 26 23:38:20.557 INFO Logging to file                         path: "/data/lighthouse/beacon-data/beacon/logs/beacon.log"
  Jun 26 23:38:20.573 INFO Lighthouse started                      version: Lighthouse/v5.2.0-f1d88ba
  Jun 26 23:38:20.573 INFO Configured for network                  name: custom (/network-configs)
  Jun 26 23:38:20.574 INFO Data directory initialised              datadir: /data/lighthouse/beacon-data
  Jun 26 23:38:20.575 WARN Discv5 packet filter is disabled
  Jun 26 23:38:20.575 WARN Ignoring --http-allow-sync-stalled      info: this flag is deprecated and will be removed
  Jun 26 23:38:20.583 INFO Deposit contract                        address: 0x4242424242424242424242424242424242424242, deploy_block: 0
  Jun 26 23:38:20.615 INFO Blob DB initialized                     oldest_blob_slot: Some(Slot(0)), path: "/data/lighthouse/beacon-data/beacon/blobs_db", service: freezer_db
  Jun 26 23:38:20.647 INFO Starting from known genesis state       service: beacon
  
  == FINISHED SERVICE 'cl-1-lighthouse-geth' LOGS ===================================
  Caused by: An error occurred while waiting for all TCP and UDP ports to be open
  Caused by: Unsuccessful ports check for IP '172.16.0.13' and port spec '{privatePortSpec:0xc000d9cb10}', even after '240' retries with '500' milliseconds in between retries. Timeout '2m0s' has been reached
  Caused by: An error occurred while calling network address '172.16.0.13:9000' with port protocol 'TCP' and using time out '200ms'
  Caused by: dial tcp 172.16.0.13:9000: i/o timeout

Error encountered running Starlark code.

⭐ us on GitHub - https://github.com/kurtosis-tech/kurtosis
INFO[2024-06-27T07:40:20+08:00] =================================================== 
INFO[2024-06-27T07:40:20+08:00] ||          Created enclave: dry-geyser          || 
INFO[2024-06-27T07:40:20+08:00] =================================================== 
Name:            dry-geyser
UUID:            8a9886d10290
Status:          RUNNING
Creation Time:   Thu, 27 Jun 2024 07:36:57 CST
Flags:           

========================================= Files Artifacts =========================================
UUID           Name
bb406afedb3a   1-lighthouse-geth-0-63-0
0c1bef28eb78   2-lighthouse-geth-64-127-0
a86050e79779   3-teku-geth-128-191-0
357236e5d3f6   el_cl_genesis_data
56a5a841d366   final-genesis-timestamp
6e53c4476eb2   genesis-el-cl-env-file
2c691f370de9   genesis_validators_root
bda9564aa987   jwt_file
e7799dd9ca68   keymanager_file
36c3d12fe1aa   prysm-password

========================================== User Services ==========================================
UUID           Name                                             Ports                                         Status
294aacfef506   el-1-geth-lighthouse                             engine-rpc: 8551/tcp -> 127.0.0.1:49326       RUNNING
                                                                metrics: 9001/tcp -> 127.0.0.1:49325          
                                                                rpc: 8545/tcp -> http://127.0.0.1:49328       
                                                                tcp-discovery: 30303/tcp -> 127.0.0.1:49324   
                                                                udp-discovery: 30303/udp -> 127.0.0.1:49187   
                                                                ws: 8546/tcp -> 127.0.0.1:49327               
21e3516db10e   el-2-geth-lighthouse                             engine-rpc: 8551/tcp -> 127.0.0.1:49331       RUNNING
                                                                metrics: 9001/tcp -> 127.0.0.1:49330          
                                                                rpc: 8545/tcp -> http://127.0.0.1:49333       
                                                                tcp-discovery: 30303/tcp -> 127.0.0.1:49329   
                                                                udp-discovery: 30303/udp -> 127.0.0.1:49188   
                                                                ws: 8546/tcp -> 127.0.0.1:49332               
1392da84cc1e   el-3-geth-teku                                   engine-rpc: 8551/tcp -> 127.0.0.1:49336       RUNNING
                                                                metrics: 9001/tcp -> 127.0.0.1:49335          
                                                                rpc: 8545/tcp -> http://127.0.0.1:49338       
                                                                tcp-discovery: 30303/tcp -> 127.0.0.1:49334   
                                                                udp-discovery: 30303/udp -> 127.0.0.1:49189   
                                                                ws: 8546/tcp -> 127.0.0.1:49337               
154ff54a408d   validator-key-generation-cl-validator-keystore   <none>                                        RUNNING

(base) root@ST-ZY-28:/var/zk_evm# 
INFO[2024-06-27T07:40:20+08:00] ===================================================
Name:            dry-geyser
UUID:            8a9886d10290
Status:          RUNNING
Creation Time:   Thu, 27 Jun 2024 07:36:57 CST
Flags:

========================================= Files Artifacts =========================================
UUID           Name
bb406afedb3a   1-lighthouse-geth-0-63-0
0c1bef28eb78   2-lighthouse-geth-64-127-0
a86050e79779   3-teku-geth-128-191-0
357236e5d3f6   el_cl_genesis_data
56a5a841d366   final-genesis-timestamp
6e53c4476eb2   genesis-el-cl-env-file
2c691f370de9   genesis_validators_root
bda9564aa987   jwt_file
e7799dd9ca68   keymanager_file
36c3d12fe1aa   prysm-password

========================================== User Services ==========================================
UUID           Name                                             Ports                                         Status
294aacfef506   el-1-geth-lighthouse                             engine-rpc: 8551/tcp -> 127.0.0.1:49326       RUNNING
                                                                metrics: 9001/tcp -> 127.0.0.1:49325
                                                                rpc: 8545/tcp -> http://127.0.0.1:49328
                                                                tcp-discovery: 30303/tcp -> 127.0.0.1:49324
                                                                udp-discovery: 30303/udp -> 127.0.0.1:49187
                                                                ws: 8546/tcp -> 127.0.0.1:49327
21e3516db10e   el-2-geth-lighthouse                             engine-rpc: 8551/tcp -> 127.0.0.1:49331       RUNNING
                                                                metrics: 9001/tcp -> 127.0.0.1:49330
                                                                rpc: 8545/tcp -> http://127.0.0.1:49333
                                                                tcp-discovery: 30303/tcp -> 127.0.0.1:49329
                                                                udp-discovery: 30303/udp -> 127.0.0.1:49188
                                                                ws: 8546/tcp -> 127.0.0.1:49332
1392da84cc1e   el-3-geth-teku                                   engine-rpc: 8551/tcp -> 127.0.0.1:49336       RUNNING
                                                                metrics: 9001/tcp -> 127.0.0.1:49335
                                                                rpc: 8545/tcp -> http://127.0.0.1:49338
                                                                tcp-discovery: 30303/tcp -> 127.0.0.1:49334
                                                                udp-discovery: 30303/udp -> 127.0.0.1:49189
                                                                ws: 8546/tcp -> 127.0.0.1:49337
154ff54a408d   validator-key-generation-cl-validator-keystore   <none>                                        RUNNING

(base) root@ST-ZY-28:/var/zk_evm# 
  
  == FINISHED SERVICE 'cl-1-lighthouse-geth' LOGS ===================================
  Caused by: An error occurred while waiting for all TCP and UDP ports to be open
  Caused by: Unsuccessful ports check for IP '172.16.0.13' and port spec '{privatePortSpec:0xc000d9cb10}', even after '240' retries with '500' milliseconds in between retries. Timeout '2m0s' has been reached
  Caused by: An error occurred while calling network address '172.16.0.13:9000' with port protocol 'TCP' and using time out '200ms'
  Caused by: dial tcp 172.16.0.13:9000: i/o timeout

Error encountered running Starlark code.

⭐ us on GitHub - https://github.com/kurtosis-tech/kurtosis
INFO[2024-06-27T07:40:20+08:00] ===================================================
INFO[2024-06-27T07:40:20+08:00] ||          Created enclave: dry-geyser          ||
INFO[2024-06-27T07:40:20+08:00] ===================================================
Name:            dry-geyser
UUID:            8a9886d10290
Status:          RUNNING
Creation Time:   Thu, 27 Jun 2024 07:36:57 CST
Flags:

========================================= Files Artifacts =========================================
UUID           Name
bb406afedb3a   1-lighthouse-geth-0-63-0
0c1bef28eb78   2-lighthouse-geth-64-127-0
a86050e79779   3-teku-geth-128-191-0
357236e5d3f6   el_cl_genesis_data
56a5a841d366   final-genesis-timestamp
6e53c4476eb2   genesis-el-cl-env-file
2c691f370de9   genesis_validators_root
bda9564aa987   jwt_file
e7799dd9ca68   keymanager_file
36c3d12fe1aa   prysm-password

========================================== User Services ==========================================
UUID           Name                                             Ports                                         Status
294aacfef506   el-1-geth-lighthouse                             engine-rpc: 8551/tcp -> 127.0.0.1:49326       RUNNING
                                                                metrics: 9001/tcp -> 127.0.0.1:49325
                                                                rpc: 8545/tcp -> http://127.0.0.1:49328
                                                                tcp-discovery: 30303/tcp -> 127.0.0.1:49324
                                                                udp-discovery: 30303/udp -> 127.0.0.1:49187
                                                                ws: 8546/tcp -> 127.0.0.1:49327
21e3516db10e   el-2-geth-lighthouse                             engine-rpc: 8551/tcp -> 127.0.0.1:49331       RUNNING
                                                                metrics: 9001/tcp -> 127.0.0.1:49330
                                                                rpc: 8545/tcp -> http://127.0.0.1:49333
                                                                tcp-discovery: 30303/tcp -> 127.0.0.1:49329
                                                                udp-discovery: 30303/udp -> 127.0.0.1:49188
                                                                ws: 8546/tcp -> 127.0.0.1:49332
1392da84cc1e   el-3-geth-teku                                   engine-rpc: 8551/tcp -> 127.0.0.1:49336       RUNNING
                                                                metrics: 9001/tcp -> 127.0.0.1:49335
                                                                rpc: 8545/tcp -> http://127.0.0.1:49338
                                                                tcp-discovery: 30303/tcp -> 127.0.0.1:49334
                                                                udp-discovery: 30303/udp -> 127.0.0.1:49189
                                                                ws: 8546/tcp -> 127.0.0.1:49337
154ff54a408d   validator-key-generation-cl-validator-keystore   <none>                                        RUNNING

my network_params2.yaml is：

participants:
  - el_type: geth
    cl_type: lighthouse
    count: 2
  - el_type: geth
    cl_type: teku
network_params:
  network_id: "585858"
additional_services:
  - dora

port is free

`

(base) root@ST-ZY-28:/var/zk_evm# sudo netstat -tuln | grep -E '9000|4000|5054'
(base) root@ST-ZY-28:/var/zk_evm# sudo ufw status
sudo ufw allow 4000/tcp
sudo ufw allow 5054/tcp
sudo ufw allow 9000/tcp
sudo ufw allow 9000/udp
Status: inactive
(base) root@ST-ZY-28:/var/zk_evm#

`
@barnabasbusa
Copy link
Contributor

Can you try to update your docker engine version to 27.0.2. And possibly try to bump your ubuntu to 24.04?

@KatyaRyazantseva
Copy link
Contributor

I have the same issue but on Mac. The latest docker engine version for mac is v.26.1.4. Do I have any chance?

MacOS: Monterey 12.7.5
Docker Desktop: 4.31.0
Docker Engine: 26.1.4
Kurtosis CLI: 0.90.1

@barnabasbusa
Copy link
Contributor

Could you check if you seeing the same issues with a different cl_type too?

e.g lodestar ?

@KatyaRyazantseva
Copy link
Contributor

I've checked lodestar, prysm, teku, nimbus, grandine. The issue occurs only for lighthouse. And I have to comment on port_publisher param. With default settings it doesn't work at all.

# port_publisher:
#   public_port_start: null
#   nat_exit_ip: KURTOSIS_IP_ADDR_PLACEHOLDER

@barnabasbusa
Copy link
Contributor

The port publisher parameters have been reworked in #677.

Can you ensure that your virtual disk has enough free space and that you running the latest of everything? Kurtosis version 1.0.0 or newer, and the main branch of this repo.

@KatyaRyazantseva
Copy link
Contributor

KatyaRyazantseva commented Jul 11, 2024 •
edited
Loading

I have 30Gb available, Kurtosis version 1.0.0 and I am running kurtosis run github.com/ethpandaops/ethereum-package. I guess it's the main brunch. I deleted all the lighthouse docker images before running. Currently I have ethpandaops/lighthouse:stable (created 14 days ago).

Here is the latest error. The port can change from launch to launch: 9000, 4000, 5054.

Adding service with name 'cl-1-lighthouse-geth' and image 'ethpandaops/lighthouse:stable'
There was an error executing Starlark code
An error occurred executing instruction (number 25) at github.com/ethpandaops/ethereum-package/src/cl/lighthouse/lighthouse_launcher.star[135:38]:
  add_service(name="cl-1-lighthouse-geth", config=ServiceConfig(image="ethpandaops/lighthouse:stable", ports={"http": PortSpec(number=4000, transport_protocol="TCP", application_protocol="http"), "metrics": PortSpec(number=5054, transport_protocol="TCP", application_protocol="http"), "tcp-discovery": PortSpec(number=9000, transport_protocol="TCP", application_protocol=""), "udp-discovery": PortSpec(number=9000, transport_protocol="UDP", application_protocol="")}, public_ports={}, files={"/jwt": "jwt_file", "/network-configs": "el_cl_genesis_data"}, cmd=["lighthouse", "beacon_node", "--debug-level=info", "--datadir=/data/lighthouse/beacon-data", "--disable-enr-auto-update", "--enr-address=KURTOSIS_IP_ADDR_PLACEHOLDER", "--enr-udp-port=9000", "--enr-tcp-port=9000", "--listen-address=0.0.0.0", "--port=9000", "--http", "--http-address=0.0.0.0", "--http-port=4000", "--http-allow-sync-stalled", "--slots-per-restore-point=32", "--disable-packet-filter", "--execution-endpoints=http://{{kurtosis:a77cdf0a62b648ffafc7b284ff43b088:ip_address.runtime_value}}:8551", "--jwt-secrets=/jwt/jwtsecret", "--suggested-fee-recipient=0x8943545177806ED17B9F23F0a21ee5948eCaa776", "--subscribe-all-subnets", "--metrics", "--metrics-address=0.0.0.0", "--metrics-allow-origin=*", "--metrics-port=5054", "--enable-private-discovery", "--testnet-dir=/network-configs"], env_vars={"RUST_BACKTRACE": "full"}, private_ip_address_placeholder="KURTOSIS_IP_ADDR_PLACEHOLDER", max_cpu=1000, min_cpu=50, max_memory=1024, min_memory=256, ready_conditions=ReadyCondition(recipe=GetHttpRequestRecipe(port_id="http", endpoint="/eth/v1/node/health"), field="code", assertion="IN", target_value=[200, 206], timeout="15m"), labels={"ethereum-package.client": "lighthouse", "ethereum-package.client-image": "ethpandaops-lighthouse_stable", "ethereum-package.client-type": "beacon", "ethereum-package.connected-client": "geth", "ethereum-package.sha256": ""}, tolerations=[], node_selectors={}))
  Caused by: Unexpected error occurred starting service 'cl-1-lighthouse-geth'
  Caused by: An error occurred waiting for all TCP and UDP ports to be open for service 'cl-1-lighthouse-geth' with private IP '172.16.4.11'; this is usually due to a misconfiguration in the service itself, so here are the logs:
  == SERVICE 'cl-1-lighthouse-geth' LOGS ===================================
  Jul 11 16:06:50.076 INFO Logging to file                         path: "/data/lighthouse/beacon-data/beacon/logs/beacon.log"
  Jul 11 16:06:50.221 INFO Lighthouse started                      version: Lighthouse/v5.2.0-f1d88ba
  Jul 11 16:06:50.222 INFO Configured for network                  name: custom (/network-configs)
  Jul 11 16:06:50.244 INFO Data directory initialised              datadir: /data/lighthouse/beacon-data
  Jul 11 16:06:50.247 WARN Discv5 packet filter is disabled
  Jul 11 16:06:50.247 WARN Ignoring --http-allow-sync-stalled      info: this flag is deprecated and will be removed
  Jul 11 16:06:50.259 INFO Deposit contract                        address: 0x4242424242424242424242424242424242424242, deploy_block: 0
  Jul 11 16:06:50.309 INFO Blob DB initialized                     oldest_blob_slot: Some(Slot(0)), path: "/data/lighthouse/beacon-data/beacon/blobs_db", service: freezer_db
  Jul 11 16:06:50.349 INFO Starting from known genesis state       service: beacon

  == FINISHED SERVICE 'cl-1-lighthouse-geth' LOGS ===================================
  Caused by: An error occurred while waiting for all TCP and UDP ports to be open
  Caused by: Unsuccessful ports check for IP '172.16.4.11' and port spec '{privatePortSpec:0xc00070db90}', even after '240' retries with '500' milliseconds in between retries. Timeout '2m0s' has been reached
  Caused by: An error occurred while calling network address '172.16.4.11:9000' with port protocol 'TCP' and using time out '200ms'
  Caused by: dial tcp 172.16.4.11:9000: i/o timeout

Error encountered running Starlark code.

@barnabasbusa
Copy link
Contributor

can you try to do cl_max_mem override to something more than 1024? Let's try with 4096.

@KatyaRyazantseva
Copy link
Contributor

What I found out, if I run kurtosis run github.com/ethpandaops/ethereum-package, I have this timeout error. But if I run the same with --args-file network_params.yaml, it works fine. Even though the remote file and the local file are identical. So, even not applying 4096 it works. Checked several times, it is reproduced every time.

@snail-inO
Copy link

I also have the same problem, and I found that the error is resolved by changing the cl_image from ethpandaops/lighthouse:stable (the default) to sigp/lighthouse:latest-unstable.
This might be because of some config mismatch between the el and cl?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@barnabasbusa @snail-inO @KatyaRyazantseva @davidLeeeeeeeeeeee