Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[bitnami/etcd] Intermittent Port 2380 Conflict Error on Pod Restart for Single-node ETCD Deployment #75295

Open
finday3 opened this issue Nov 29, 2024 · 2 comments
Open

[bitnami/etcd] Intermittent Port 2380 Conflict Error on Pod Restart for Single-node ETCD Deployment #75295

finday3 opened this issue Nov 29, 2024 · 2 comments
Assignees
@migruiz4
Labels
etcd in-progress tech-issues The user has a technical issue about an application

Comments

@finday3
Copy link

finday3 commented Nov 29, 2024 •
edited
Loading

Name and Version

bitnami/etcd:3.5.14

What architecture are you using?

amd64

What steps will reproduce the bug?

  1. Deploy a single-node ETCD using the Bitnami Etcd Chart.
  2. Allow it to run for a period of time.
  3. Execute a POD restart.
  4. Occasionally, observe the error indicating that port 2380 is already in use.

What is the expected behavior?

ETCD should start normally after a POD restart without encountering port conflicts.

What do you see instead?

Intermittently, ETCD fails to start with the error: listen tcp 0.0.0.0:2380: bind: address already in use. This issue does not occur consistently.

Additional information

Logs:
In cases where ETCD starts successfully, the logs contain the following snippet:

etcd 10:10:20.96 INFO ==> Stopping etcd
Terminated
etcd 10:10:22.01 INFO ==> ** etcd setup finished! **

etcd 10:10:22.10 INFO ==> ** Starting etcd **

Here is a snippet of the failed logs:

etcd 09:58:22.86 INFO  ==> Welcome to the Bitnami etcd container
etcd 09:58:22.87 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
etcd 09:58:22.87 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
etcd 09:58:22.87 INFO  ==> Upgrade to Tanzu Application Catalog for production environments to access custom-configured and pre-packaged software components. Gain enhanced features, including Software Bill of Materials (SBOM), CVE scan result reports, and VEX documents. To learn more, visit https://bitnami.com/enterprise
etcd 09:58:22.88 INFO  ==> 
etcd 09:58:23.06 INFO  ==> ** Starting etcd setup **
etcd 09:58:23.10 INFO  ==> Validating settings in ETCD_* env vars..
etcd 09:58:23.13 INFO  ==> Initializing etcd
etcd 09:58:23.13 INFO  ==> Generating etcd config file using env variables
etcd 09:58:23.30 INFO  ==> Detected data from previous deployments
etcd 09:58:23.36 INFO  ==> Starting etcd in background
{"level":"info","ts":"2024-11-29T09:58:23.477997+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_ADVERTISE_CLIENT_URLS","variable-value":"http://ucp-etcd-0.ucp-etcd-headless.service-software.svc.cluster.local:2379,http://ucp-etcd.service-software.svc.cluster.local:2379"}
{"level":"info","ts":"2024-11-29T09:58:23.4782+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTH_TOKEN","variable-value":"jwt,priv-key=/opt/bitnami/etcd/certs/token/jwt-token.pem,sign-method=RS256,ttl=10m"}
{"level":"info","ts":"2024-11-29T09:58:23.478221+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTO_COMPACTION_MODE","variable-value":"periodic"}
{"level":"info","ts":"2024-11-29T09:58:23.478232+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTO_COMPACTION_RETENTION","variable-value":"1"}
{"level":"info","ts":"2024-11-29T09:58:23.478246+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTO_TLS","variable-value":"false"}
{"level":"info","ts":"2024-11-29T09:58:23.478262+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_CLIENT_CERT_AUTH","variable-value":"false"}
{"level":"info","ts":"2024-11-29T09:58:23.478277+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_DATA_DIR","variable-value":"/bitnami/etcd/data"}
{"level":"info","ts":"2024-11-29T09:58:23.478292+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_ELECTION_TIMEOUT","variable-value":"5000"}
{"level":"info","ts":"2024-11-29T09:58:23.478346+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_HEARTBEAT_INTERVAL","variable-value":"1000"}
{"level":"info","ts":"2024-11-29T09:58:23.478369+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_ADVERTISE_PEER_URLS","variable-value":"http://ucp-etcd-0.ucp-etcd-headless.service-software.svc.cluster.local:2380"}
{"level":"info","ts":"2024-11-29T09:58:23.478388+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LISTEN_CLIENT_URLS","variable-value":"http://0.0.0.0:2379"}
{"level":"info","ts":"2024-11-29T09:58:23.478401+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LISTEN_PEER_URLS","variable-value":"http://0.0.0.0:2380"}
{"level":"info","ts":"2024-11-29T09:58:23.478415+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LOG_LEVEL","variable-value":"debug"}
{"level":"info","ts":"2024-11-29T09:58:23.478427+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_MAX_REQUEST_BYTES","variable-value":"5242880"}
{"level":"info","ts":"2024-11-29T09:58:23.478439+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_NAME","variable-value":"ucp-etcd-0"}
{"level":"info","ts":"2024-11-29T09:58:23.478447+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_PEER_AUTO_TLS","variable-value":"false"}
{"level":"info","ts":"2024-11-29T09:58:23.478464+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_QUOTA_BACKEND_BYTES","variable-value":"8589934592"}
{"level":"warn","ts":"2024-11-29T09:58:23.478527+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_TRUSTED_CA_FILE="}
{"level":"warn","ts":"2024-11-29T09:58:23.478555+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DISABLE_STORE_MEMBER_ID=no"}
{"level":"warn","ts":"2024-11-29T09:58:23.478565+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CONF_FILE=/opt/bitnami/etcd/conf/etcd.yaml"}
{"level":"warn","ts":"2024-11-29T09:58:23.478577+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_SNAPSHOT_HISTORY_LIMIT=1"}
{"level":"warn","ts":"2024-11-29T09:58:23.478608+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_ON_K8S=yes"}
{"level":"warn","ts":"2024-11-29T09:58:23.478617+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_SNAPSHOTS_DIR=/snapshots"}
{"level":"warn","ts":"2024-11-29T09:58:23.478629+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_BIN_DIR=/opt/bitnami/etcd/bin"}
{"level":"warn","ts":"2024-11-29T09:58:23.47864+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_VOLUME_DIR=/bitnami/etcd"}
{"level":"warn","ts":"2024-11-29T09:58:23.478649+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_INITIAL_CLUSTER_TOKEN="}
{"level":"warn","ts":"2024-11-29T09:58:23.47866+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DEFAULT_CONF_DIR=/opt/bitnami/etcd/conf.default"}
{"level":"warn","ts":"2024-11-29T09:58:23.478681+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_ROOT_PASSWORD=DB-pwd@123"}
{"level":"warn","ts":"2024-11-29T09:58:23.478696+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CLUSTER_DOMAIN=ucp-etcd-headless.service-software.svc.cluster.local"}
{"level":"warn","ts":"2024-11-29T09:58:23.478705+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DISASTER_RECOVERY=no"}
{"level":"warn","ts":"2024-11-29T09:58:23.478715+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_KEY_FILE="}
{"level":"warn","ts":"2024-11-29T09:58:23.478724+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CONF_DIR=/opt/bitnami/etcd/conf"}
{"level":"warn","ts":"2024-11-29T09:58:23.478743+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DAEMON_GROUP=etcd"}
{"level":"warn","ts":"2024-11-29T09:58:23.478752+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_START_FROM_SNAPSHOT=no"}
{"level":"warn","ts":"2024-11-29T09:58:23.478767+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_INIT_SNAPSHOT_FILENAME="}
{"level":"warn","ts":"2024-11-29T09:58:23.478785+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_INIT_SNAPSHOTS_DIR=/init-snapshot"}
{"level":"warn","ts":"2024-11-29T09:58:23.478799+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DISABLE_PRESTOP=no"}
{"level":"warn","ts":"2024-11-29T09:58:23.478811+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_TMP_DIR=/opt/bitnami/etcd/tmp"}
{"level":"warn","ts":"2024-11-29T09:58:23.478821+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_INITIAL_CLUSTER_STATE="}
{"level":"warn","ts":"2024-11-29T09:58:23.478829+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_BASE_DIR=/opt/bitnami/etcd"}
{"level":"warn","ts":"2024-11-29T09:58:23.478837+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_INITIAL_CLUSTER="}
{"level":"warn","ts":"2024-11-29T09:58:23.478862+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CERT_FILE="}
{"level":"warn","ts":"2024-11-29T09:58:23.478873+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_NEW_MEMBERS_ENV_FILE=/bitnami/etcd/data/new_member_envs"}
{"level":"warn","ts":"2024-11-29T09:58:23.478881+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DAEMON_USER=etcd"}
{"level":"warn","ts":"2024-11-29T09:58:23.478995+0800","caller":"embed/config.go:687","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"info","ts":"2024-11-29T09:58:23.479038+0800","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["etcd"]}
{"level":"warn","ts":"2024-11-29T09:58:23.479125+0800","caller":"etcdmain/etcd.go:446","msg":"found invalid file under data directory","filename":"member_id","data-dir":"/bitnami/etcd/data"}
{"level":"info","ts":"2024-11-29T09:58:23.479135+0800","caller":"etcdmain/etcd.go:116","msg":"server has been already initialized","data-dir":"/bitnami/etcd/data","dir-type":"member"}
{"level":"warn","ts":"2024-11-29T09:58:23.479151+0800","caller":"embed/config.go:687","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"info","ts":"2024-11-29T09:58:23.47918+0800","caller":"embed/etcd.go:127","msg":"configuring peer listeners","listen-peer-urls":["http://0.0.0.0:2380"]}
{"level":"info","ts":"2024-11-29T09:58:23.479453+0800","caller":"embed/etcd.go:135","msg":"configuring client listeners","listen-client-urls":["http://0.0.0.0:2379"]}
{"level":"info","ts":"2024-11-29T09:58:23.479584+0800","caller":"embed/etcd.go:309","msg":"starting an etcd server","etcd-version":"3.5.14","git-sha":"bf51a53","go-version":"go1.21.10","go-os":"linux","go-arch":"amd64","max-cpu-set":32,"max-cpu-available":32,"member-initialized":true,"name":"ucp-etcd-0","data-dir":"/bitnami/etcd/data","wal-dir":"","wal-dir-dedicated":"","member-dir":"/bitnami/etcd/data/member","force-new-cluster":false,"heartbeat-interval":"1s","election-timeout":"5s","initial-election-tick-advance":true,"snapshot-count":100000,"max-wals":5,"max-snapshots":5,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://ucp-etcd-0.ucp-etcd-headless.service-software.svc.cluster.local:2380"],"listen-peer-urls":["http://0.0.0.0:2380"],"advertise-client-urls":["http://ucp-etcd-0.ucp-etcd-headless.service-software.svc.cluster.local:2379","http://ucp-etcd.service-software.svc.cluster.local:2379"],"listen-client-urls":["http://0.0.0.0:2379"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-backend-bytes":8589934592,"max-request-bytes":5242880,"max-concurrent-streams":4294967295,"pre-vote":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","compact-check-time-enabled":false,"compact-check-time-interval":"1m0s","auto-compaction-mode":"periodic","auto-compaction-retention":"1h0m0s","auto-compaction-interval":"1h0m0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
{"level":"info","ts":"2024-11-29T09:58:33.486782+0800","caller":"etcdserver/backend.go:85","msg":"db file is flocked by another process, or taking too long","path":"/bitnami/etcd/data/member/snap/db","took":"10.006867816s"}
{"level":"warn","ts":"2024-11-29T09:58:42.795397+0800","caller":"wal/util.go:90","msg":"ignored file in WAL directory","path":"0000000000000035-0000000001cd69a3.wal.broken"}
{"level":"info","ts":"2024-11-29T09:59:01.113223+0800","caller":"etcdserver/server.go:511","msg":"recovered v2 store from snapshot","snapshot-index":38500444,"snapshot-size":"15 kB"}
{"level":"info","ts":"2024-11-29T09:59:01.113337+0800","caller":"etcdserver/server.go:524","msg":"recovered v3 backend from snapshot","backend-size-bytes":1453449216,"backend-size":"1.5 GB","backend-size-in-use-bytes":1302470656,"backend-size-in-use":"1.3 GB"}
{"level":"warn","ts":"2024-11-29T09:59:01.113465+0800","caller":"wal/util.go:90","msg":"ignored file in WAL directory","path":"0000000000000035-0000000001cd69a3.wal.broken"}
{"level":"info","ts":"2024-11-29T09:59:28.649315+0800","caller":"etcdserver/raft.go:530","msg":"restarting local member","cluster-id":"23dba460e402a254","local-member-id":"ef342e15cd67e473","commit-index":38594553}
{"level":"info","ts":"2024-11-29T09:59:28.661461+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"ef342e15cd67e473 switched to configuration voters=(17236452345029059699)"}
{"level":"info","ts":"2024-11-29T09:59:28.661564+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"ef342e15cd67e473 became follower at term 15"}
{"level":"info","ts":"2024-11-29T09:59:28.661582+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft ef342e15cd67e473 [peers: [ef342e15cd67e473], term: 15, commit: 38594553, applied: 38500444, lastindex: 38594553, lastterm: 15]"}
{"level":"info","ts":"2024-11-29T09:59:28.710075+0800","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"}
{"level":"info","ts":"2024-11-29T09:59:28.710125+0800","caller":"membership/cluster.go:278","msg":"recovered/added member from store","cluster-id":"23dba460e402a254","local-member-id":"ef342e15cd67e473","recovered-remote-peer-id":"ef342e15cd67e473","recovered-remote-peer-urls":["http://ucp-etcd-0.ucp-etcd-headless.service-software.svc.cluster.local:2380"]}
{"level":"info","ts":"2024-11-29T09:59:28.710142+0800","caller":"membership/cluster.go:287","msg":"set cluster version from store","cluster-version":"3.5"}
{"level":"info","ts":"2024-11-29T09:59:28.770875+0800","caller":"mvcc/kvstore.go:341","msg":"restored last compact revision","meta-bucket-name":"meta","meta-bucket-name-key":"finishedCompactRev","restored-compact-revision":670902}
etcd 10:00:27.50 INFO  ==> ** etcd setup finished! **

etcd 10:00:27.57 INFO  ==> ** Starting etcd **
{"level":"info","ts":"2024-11-29T10:00:28.009572+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_ADVERTISE_CLIENT_URLS","variable-value":"http://ucp-etcd-0.ucp-etcd-headless.service-software.svc.cluster.local:2379,http://ucp-etcd.service-software.svc.cluster.local:2379"}
{"level":"info","ts":"2024-11-29T10:00:28.009788+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTH_TOKEN","variable-value":"jwt,priv-key=/opt/bitnami/etcd/certs/token/jwt-token.pem,sign-method=RS256,ttl=10m"}
{"level":"info","ts":"2024-11-29T10:00:28.009805+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTO_COMPACTION_MODE","variable-value":"periodic"}
{"level":"info","ts":"2024-11-29T10:00:28.00982+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTO_COMPACTION_RETENTION","variable-value":"1"}
{"level":"info","ts":"2024-11-29T10:00:28.009831+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTO_TLS","variable-value":"false"}
{"level":"info","ts":"2024-11-29T10:00:28.009847+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_CLIENT_CERT_AUTH","variable-value":"false"}
{"level":"info","ts":"2024-11-29T10:00:28.009863+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_DATA_DIR","variable-value":"/bitnami/etcd/data"}
{"level":"info","ts":"2024-11-29T10:00:28.009878+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_ELECTION_TIMEOUT","variable-value":"5000"}
{"level":"info","ts":"2024-11-29T10:00:28.009928+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_HEARTBEAT_INTERVAL","variable-value":"1000"}
{"level":"info","ts":"2024-11-29T10:00:28.009948+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_ADVERTISE_PEER_URLS","variable-value":"http://ucp-etcd-0.ucp-etcd-headless.service-software.svc.cluster.local:2380"}
{"level":"info","ts":"2024-11-29T10:00:28.00998+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LISTEN_CLIENT_URLS","variable-value":"http://0.0.0.0:2379"}
{"level":"info","ts":"2024-11-29T10:00:28.010003+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LISTEN_PEER_URLS","variable-value":"http://0.0.0.0:2380"}
{"level":"info","ts":"2024-11-29T10:00:28.010018+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LOG_LEVEL","variable-value":"debug"}
{"level":"info","ts":"2024-11-29T10:00:28.010032+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_MAX_REQUEST_BYTES","variable-value":"5242880"}
{"level":"info","ts":"2024-11-29T10:00:28.01005+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_NAME","variable-value":"ucp-etcd-0"}
{"level":"info","ts":"2024-11-29T10:00:28.010068+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_PEER_AUTO_TLS","variable-value":"false"}
{"level":"info","ts":"2024-11-29T10:00:28.010096+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_QUOTA_BACKEND_BYTES","variable-value":"8589934592"}
{"level":"warn","ts":"2024-11-29T10:00:28.010179+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_TRUSTED_CA_FILE="}
{"level":"warn","ts":"2024-11-29T10:00:28.010228+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DISABLE_STORE_MEMBER_ID=no"}
{"level":"warn","ts":"2024-11-29T10:00:28.010238+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CONF_FILE=/opt/bitnami/etcd/conf/etcd.yaml"}
{"level":"warn","ts":"2024-11-29T10:00:28.010248+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_SNAPSHOT_HISTORY_LIMIT=1"}
{"level":"warn","ts":"2024-11-29T10:00:28.010289+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_ON_K8S=yes"}
{"level":"warn","ts":"2024-11-29T10:00:28.010299+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_SNAPSHOTS_DIR=/snapshots"}
{"level":"warn","ts":"2024-11-29T10:00:28.010315+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_BIN_DIR=/opt/bitnami/etcd/bin"}
{"level":"warn","ts":"2024-11-29T10:00:28.010327+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_VOLUME_DIR=/bitnami/etcd"}
{"level":"warn","ts":"2024-11-29T10:00:28.010335+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_INITIAL_CLUSTER_TOKEN="}
{"level":"warn","ts":"2024-11-29T10:00:28.010347+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DEFAULT_CONF_DIR=/opt/bitnami/etcd/conf.default"}
{"level":"warn","ts":"2024-11-29T10:00:28.010372+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CLUSTER_DOMAIN=ucp-etcd-headless.service-software.svc.cluster.local"}
{"level":"warn","ts":"2024-11-29T10:00:28.010382+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DISASTER_RECOVERY=no"}
{"level":"warn","ts":"2024-11-29T10:00:28.010391+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_KEY_FILE="}
{"level":"warn","ts":"2024-11-29T10:00:28.010399+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CONF_DIR=/opt/bitnami/etcd/conf"}
{"level":"warn","ts":"2024-11-29T10:00:28.010419+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DAEMON_GROUP=etcd"}
{"level":"warn","ts":"2024-11-29T10:00:28.010429+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_START_FROM_SNAPSHOT=no"}
{"level":"warn","ts":"2024-11-29T10:00:28.010444+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_INIT_SNAPSHOT_FILENAME="}
{"level":"warn","ts":"2024-11-29T10:00:28.010454+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_INIT_SNAPSHOTS_DIR=/init-snapshot"}
{"level":"warn","ts":"2024-11-29T10:00:28.010465+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DISABLE_PRESTOP=no"}
{"level":"warn","ts":"2024-11-29T10:00:28.010475+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_TMP_DIR=/opt/bitnami/etcd/tmp"}
{"level":"warn","ts":"2024-11-29T10:00:28.010485+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_INITIAL_CLUSTER_STATE="}
{"level":"warn","ts":"2024-11-29T10:00:28.010492+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_BASE_DIR=/opt/bitnami/etcd"}
{"level":"warn","ts":"2024-11-29T10:00:28.010505+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_INITIAL_CLUSTER="}
{"level":"warn","ts":"2024-11-29T10:00:28.01053+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CERT_FILE="}
{"level":"warn","ts":"2024-11-29T10:00:28.01054+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_NEW_MEMBERS_ENV_FILE=/bitnami/etcd/data/new_member_envs"}
{"level":"warn","ts":"2024-11-29T10:00:28.010548+0800","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DAEMON_USER=etcd"}
{"level":"warn","ts":"2024-11-29T10:00:28.010699+0800","caller":"embed/config.go:687","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"info","ts":"2024-11-29T10:00:28.01073+0800","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["etcd"]}
{"level":"warn","ts":"2024-11-29T10:00:28.010818+0800","caller":"etcdmain/etcd.go:446","msg":"found invalid file under data directory","filename":"member_id","data-dir":"/bitnami/etcd/data"}
{"level":"info","ts":"2024-11-29T10:00:28.010831+0800","caller":"etcdmain/etcd.go:116","msg":"server has been already initialized","data-dir":"/bitnami/etcd/data","dir-type":"member"}
{"level":"warn","ts":"2024-11-29T10:00:28.010845+0800","caller":"embed/config.go:687","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"info","ts":"2024-11-29T10:00:28.010857+0800","caller":"embed/etcd.go:127","msg":"configuring peer listeners","listen-peer-urls":["http://0.0.0.0:2380"]}
{"level":"error","ts":"2024-11-29T10:00:28.011043+0800","caller":"embed/etcd.go:537","msg":"creating peer listener failed","error":"listen tcp 0.0.0.0:2380: bind: address already in use","stacktrace":"go.etcd.io/etcd/server/v3/embed.configurePeerListeners\n\tgo.etcd.io/etcd/server/v3/embed/etcd.go:537\ngo.etcd.io/etcd/server/v3/embed.StartEtcd\n\tgo.etcd.io/etcd/server/v3/embed/etcd.go:131\ngo.etcd.io/etcd/server/v3/etcdmain.startEtcd\n\tgo.etcd.io/etcd/server/v3/etcdmain/etcd.go:228\ngo.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\tgo.etcd.io/etcd/server/v3/etcdmain/etcd.go:123\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\tgo.etcd.io/etcd/server/v3/etcdmain/main.go:40\nmain.main\n\tgo.etcd.io/etcd/server/v3/main.go:31\nruntime.main\n\truntime/proc.go:267"}
{"level":"info","ts":"2024-11-29T10:00:28.011113+0800","caller":"embed/etcd.go:376","msg":"closing etcd server","name":"ucp-etcd-0","data-dir":"/bitnami/etcd/data","advertise-peer-urls":["http://ucp-etcd-0.ucp-etcd-headless.service-software.svc.cluster.local:2380"],"advertise-client-urls":["http://ucp-etcd-0.ucp-etcd-headless.service-software.svc.cluster.local:2379","http://ucp-etcd.service-software.svc.cluster.local:2379"]}
{"level":"info","ts":"2024-11-29T10:00:28.011124+0800","caller":"embed/etcd.go:378","msg":"closed etcd server","name":"ucp-etcd-0","data-dir":"/bitnami/etcd/data","advertise-peer-urls":["http://ucp-etcd-0.ucp-etcd-headless.service-software.svc.cluster.local:2380"],"advertise-client-urls":["http://ucp-etcd-0.ucp-etcd-headless.service-software.svc.cluster.local:2379","http://ucp-etcd.service-software.svc.cluster.local:2379"]}
{"level":"fatal","ts":"2024-11-29T10:00:28.011139+0800","caller":"etcdmain/etcd.go:204","msg":"discovery failed","error":"listen tcp 0.0.0.0:2380: bind: address already in use","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\tgo.etcd.io/etcd/server/v3/etcdmain/etcd.go:204\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\tgo.etcd.io/etcd/server/v3/etcdmain/main.go:40\nmain.main\n\tgo.etcd.io/etcd/server/v3/main.go:31\nruntime.main\n\truntime/proc.go:267"}```
@finday3 finday3 added the tech-issues The user has a technical issue about an application label Nov 29, 2024
@github-actions github-actions bot added the triage Triage is needed label Nov 29, 2024
@javsalgar javsalgar changed the title Intermittent Port 2380 Conflict Error on Pod Restart for Single-node ETCD Deployment [bitnami/etcd] Intermittent Port 2380 Conflict Error on Pod Restart for Single-node ETCD Deployment Nov 29, 2024
@github-actions github-actions bot removed the triage Triage is needed label Nov 29, 2024
@github-actions github-actions bot assigned migruiz4 and unassigned javsalgar Nov 29, 2024
@migruiz4
Copy link
Member

I'm sorry @finday3 but I haven't been able to reproduce your issue.

Freshly installed the latest version of the bitnami/etcd chart with default values, had it run for ~15min, then restarted the pod:

$ k get pods
NAME     READY   STATUS    RESTARTS   AGE
etcd-0   1/1     Running   0          17m
$ k delete pod etcd-0 
pod "etcd-0" deleted
$ k get pods
NAME     READY   STATUS    RESTARTS   AGE
etcd-0   0/1     Running   0          2s

In my case, the pod was able to start without issues, no bind errors in the logs:

$ k get pods
NAME     READY   STATUS    RESTARTS   AGE
etcd-0   1/1     Running   0          2m19s

Could you please provide more information about your deployment? Are you using any custom values?

@finday3
Copy link
Author

finday3 commented Dec 2, 2024 •
edited
Loading

I'm sorry @finday3 but I haven't been able to reproduce your issue.

Freshly installed the latest version of the bitnami/etcd chart with default values, had it run for ~15min, then restarted the pod:

$ k get pods
NAME     READY   STATUS    RESTARTS   AGE
etcd-0   1/1     Running   0          17m
$ k delete pod etcd-0 
pod "etcd-0" deleted
$ k get pods
NAME     READY   STATUS    RESTARTS   AGE
etcd-0   0/1     Running   0          2s

In my case, the pod was able to start without issues, no bind errors in the logs:

$ k get pods
NAME     READY   STATUS    RESTARTS   AGE
etcd-0   1/1     Running   0          2m19s

Could you please provide more information about your deployment? Are you using any custom values?

Thank you for your response. Here is the deployment information:

[root@redhatxc ~]# kubectl describe po xxx-etcd-0
Name:         xxx-etcd-0
Namespace:    test
Priority:     0
Node:         redhatxc/192.167.252.27
Start Time:   Fri, 27 Sep 2024 11:11:26 +0800
Labels:       app.kubernetes.io/component=etcd
              app.kubernetes.io/instance=xxx-etcd
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=etcd
              app.kubernetes.io/version=3.5.13
              controller-revision-hash=xxx-etcd-675b575464
              helm.sh/chart=etcd-10.0.3
              statefulset.kubernetes.io/pod-name=xxx-etcd-0
Annotations:  checksum/token-secret: 0d72490b25f9756d6e4a747be37dd246054a8c11bc5aa3600270879826491923
              container.seccomp.security.alpha.kubernetes.io/etcd: runtime/default
              k8s.v1.cni.cncf.io/networks-status:
                [{
                    "name": "k8s-pod-network",
                    "ips": [
                        "177.177.176.45",
                        "fd00:177:177:0:d6b:8ae5:2994:a713"
                    ],
                    "default": true,
                    "dns": {}
                }]
Status:       Running
IP:           177.177.176.45
IPs:
  IP:           177.177.176.45
  IP:           fd00:177:177:0:d6b:8ae5:2994:a713
Controlled By:  StatefulSet/xxx-etcd
Containers:
  etcd:
    Container ID:   docker://222d5a2c28f46530f7de6b2e79abac122a8bc19108a0ce6e7f577dcaab5c28b9
    Image:          etcd:3.5.14
    Image ID:       docker-pullable://etcd@sha256:2fb74a817f6fb9186b83d0cdce7db6c7b06c7717a8c8acc1d5692c5a61e2a3da
    Ports:          2379/TCP, 2380/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Sun, 01 Dec 2024 09:18:55 +0800
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Sun, 01 Dec 2024 09:10:05 +0800
      Finished:     Sun, 01 Dec 2024 09:13:33 +0800
    Ready:          True
    Restart Count:  3
    Limits:
      cpu:                1500m
      ephemeral-storage:  1Gi
      memory:             3Gi
    Requests:
      cpu:                1
      ephemeral-storage:  50Mi
      memory:             2Gi
    Liveness:             exec [/opt/bitnami/scripts/etcd/healthcheck.sh] delay=60s timeout=5s period=30s #success=1 #failure=5
    Readiness:            exec [/opt/bitnami/scripts/etcd/healthcheck.sh] delay=60s timeout=5s period=10s #success=1 #failure=5
    Environment:
      BITNAMI_DEBUG:                     true
      MY_POD_IP:                          (v1:status.podIP)
      MY_POD_NAME:                       xxx-etcd-0 (v1:metadata.name)
      MY_STS_NAME:                       xxx-etcd
      ETCDCTL_API:                       3
      ETCD_ON_K8S:                       yes
      ETCD_START_FROM_SNAPSHOT:          no
      ETCD_DISASTER_RECOVERY:            no
      ETCD_NAME:                         $(MY_POD_NAME)
      ETCD_DATA_DIR:                     /bitnami/etcd/data
      ETCD_LOG_LEVEL:                    debug
      ALLOW_NONE_AUTHENTICATION:         no
      ETCD_ROOT_PASSWORD:                <set to the key 'etcd-root-password' in secret 'xxx-etcd'>  Optional: false
      ETCD_AUTH_TOKEN:                   jwt,priv-key=/opt/bitnami/etcd/certs/token/jwt-token.pem,sign-method=RS256,ttl=10m
      ETCD_ADVERTISE_CLIENT_URLS:        http://$(MY_POD_NAME).xxx-etcd-headless.test.svc.cluster.local:2379,http://xxx-etcd.test.svc.cluster.local:2379
      ETCD_LISTEN_CLIENT_URLS:           http://0.0.0.0:2379
      ETCD_INITIAL_ADVERTISE_PEER_URLS:  http://$(MY_POD_NAME).xxx-etcd-headless.test.svc.cluster.local:2380
      ETCD_LISTEN_PEER_URLS:             http://0.0.0.0:2380
      ETCD_AUTO_COMPACTION_MODE:         periodic
      ETCD_AUTO_COMPACTION_RETENTION:    1
      ETCD_CLUSTER_DOMAIN:               xxx-etcd-headless.test.svc.cluster.local
      ETCD_HEARTBEAT_INTERVAL:           1000
      ETCD_ELECTION_TIMEOUT:             5000
      ETCD_QUOTA_BACKEND_BYTES:          8589934592
      ETCD_MAX_REQUEST_BYTES:            5242880
    Mounts:
      /bitnami/etcd from data (rw)
      /opt/bitnami/etcd/certs/token/ from etcd-jwt-token (ro)
      /opt/bitnami/etcd/conf/ from empty-dir (rw,path="app-conf-dir")
      /tmp from empty-dir (rw,path="tmp-dir")
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-xxx-etcd-0
    ReadOnly:   false
  empty-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  etcd-jwt-token:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  xxx-etcd-jwt-token
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 180s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 180s

We have modified the image to use the openEuler base image, while keeping the contents consistent with Bitnami. The issue does not occur consistently, and out of several dozen tests, it appeared only three times.

In this latest occurrence, the logs indicate a larger amount of data in etcd. Could there be any correlation with the data volume? Any insights or suggestions you could provide would be greatly appreciated.

Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@migruiz4 migruiz4

Labels
etcd in-progress tech-issues The user has a technical issue about an application
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@javsalgar @finday3 @migruiz4