2023-08-10 01:09:24,391 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started SLF4J: A number (3) of logging calls during the initialization phase have been intercepted and are SLF4J: now being replayed. These are subject to the filtering rules of the underlying logging system. SLF4J: See also https://www.slf4j.org/codes.html#replay 2023-08-10 01:09:24,470 DEBUG akka.event.EventStream - logger log1-Slf4jLogger started 2023-08-10 01:09:24,473 DEBUG akka.event.EventStream - Default Loggers started 2023-08-10 01:09:25,080 DEBUG a.s.Serialization(akka://kafka-lag-exporter) akka.serialization.Serialization(akka://kafka-lag-exporter) - Replacing JavaSerializer with DisabledJavaSerializer, due to `akka.actor.allow-java-serialization = off`. 2023-08-10 01:09:25,571 INFO c.l.k.KafkaClusterManager$ akka://kafka-lag-exporter/user - Starting Kafka Lag Exporter with configuration: Poll interval: 30 seconds Memory Lookup Table: Size: 60 Metrics whitelist: [.*] Admin client consumer group id: kafkalagexporter Kafka client timeout: 10 seconds Kafka retries: 0 Prometheus: port: 8000 Statically defined Clusters: Cluster name: kafka-messaging Cluster Kafka bootstrap brokers: b-1.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098,b-2.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098,b-3.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098 Consumer group whitelist: [.*] Consumer group blacklist: [] Topic whitelist: [.*] Topic blacklist: [] Watchers: Strimzi: false 2023-08-10 01:09:26,880 INFO c.l.k.KafkaClusterManager$ akka://kafka-lag-exporter/user - Cluster Added: Cluster name: kafka-messaging Cluster Kafka bootstrap brokers: b-1.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098,b-2.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098,b-3.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098 Consumer group whitelist: [.*] Consumer group blacklist: [] Topic whitelist: [.*] Topic blacklist: [] 2023-08-10 01:09:26,974 INFO c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Spawned ConsumerGroupCollector for cluster: kafka-messaging 2023-08-10 01:09:27,472 INFO o.a.k.c.consumer.ConsumerConfig akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - ConsumerConfig values: allow.auto.create.topics = true auto.commit.interval.ms = 5000 auto.offset.reset = latest bootstrap.servers = [b-1.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098, b-2.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098, b-3.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098] check.crcs = true client.dns.lookup = use_all_dns_ips client.id = consumer-kafkalagexporter-1 client.rack = connections.max.idle.ms = 540000 default.api.timeout.ms = 60000 enable.auto.commit = false exclude.internal.topics = true fetch.max.bytes = 52428800 fetch.max.wait.ms = 500 fetch.min.bytes = 1 group.id = kafkalagexporter group.instance.id = null heartbeat.interval.ms = 3000 interceptor.classes = [] internal.leave.group.on.close = true internal.throw.on.fetch.stable.offset.unsupported = false isolation.level = read_uncommitted key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer max.partition.fetch.bytes = 1048576 max.poll.interval.ms = 300000 max.poll.records = 500 metadata.max.age.ms = 300000 metric.reporters = [] metrics.num.samples = 2 metrics.recording.level = INFO metrics.sample.window.ms = 30000 partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor, class org.apache.kafka.clients.consumer.CooperativeStickyAssignor] receive.buffer.bytes = 65536 reconnect.backoff.max.ms = 1000 reconnect.backoff.ms = 50 request.timeout.ms = 10000 retry.backoff.ms = 1000 sasl.client.callback.handler.class = class software.amazon.msk.auth.iam.IAMClientCallbackHandler sasl.jaas.config = [hidden] sasl.kerberos.kinit.cmd = /usr/bin/kinit sasl.kerberos.min.time.before.relogin = 60000 sasl.kerberos.service.name = null sasl.kerberos.ticket.renew.jitter = 0.05 sasl.kerberos.ticket.renew.window.factor = 0.8 sasl.login.callback.handler.class = null sasl.login.class = null sasl.login.connect.timeout.ms = null sasl.login.read.timeout.ms = null sasl.login.refresh.buffer.seconds = 300 sasl.login.refresh.min.period.seconds = 60 sasl.login.refresh.window.factor = 0.8 sasl.login.refresh.window.jitter = 0.05 sasl.login.retry.backoff.max.ms = 10000 sasl.login.retry.backoff.ms = 100 sasl.mechanism = AWS_MSK_IAM sasl.oauthbearer.clock.skew.seconds = 30 sasl.oauthbearer.expected.audience = null sasl.oauthbearer.expected.issuer = null sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000 sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000 sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100 sasl.oauthbearer.jwks.endpoint.url = null sasl.oauthbearer.scope.claim.name = scope sasl.oauthbearer.sub.claim.name = sub sasl.oauthbearer.token.endpoint.url = null security.protocol = SASL_SSL security.providers = null send.buffer.bytes = 131072 session.timeout.ms = 45000 socket.connection.setup.timeout.max.ms = 30000 socket.connection.setup.timeout.ms = 10000 ssl.cipher.suites = null ssl.enabled.protocols = [TLSv1.2, TLSv1.3] ssl.endpoint.identification.algorithm = https ssl.engine.factory.class = null ssl.key.password = null ssl.keymanager.algorithm = SunX509 ssl.keystore.certificate.chain = null ssl.keystore.key = null ssl.keystore.location = null ssl.keystore.password = null ssl.keystore.type = JKS ssl.protocol = TLSv1.3 ssl.provider = null ssl.secure.random.implementation = null ssl.trustmanager.algorithm = PKIX ssl.truststore.certificates = null ssl.truststore.location = null ssl.truststore.password = null ssl.truststore.type = JKS value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer 2023-08-10 01:09:27,475 DEBUG o.a.k.clients.consumer.KafkaConsumer akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [Consumer clientId=consumer-kafkalagexporter-1, groupId=kafkalagexporter] Initializing the Kafka consumer 2023-08-10 01:09:28,289 DEBUG s.a.m.a.i.i.MSKCredentialProvider akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Number of options to configure credential provider 0 2023-08-10 01:09:29,073 DEBUG s.amazon.msk.auth.iam.IAMLoginModule akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - IAMLoginModule initialized 2023-08-10 01:09:29,077 INFO o.a.k.c.s.a.AbstractLogin akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Successfully logged in. 2023-08-10 01:09:30,278 DEBUG o.a.k.c.s.s.DefaultSslEngineFactory akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Created SSL context with keystore null, truststore null, provider SunJSSE. 2023-08-10 01:09:30,968 INFO o.a.kafka.common.utils.AppInfoParser akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Kafka version: 3.2.3 2023-08-10 01:09:30,969 INFO o.a.kafka.common.utils.AppInfoParser akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Kafka commitId: 50029d3ed8ba576f 2023-08-10 01:09:30,969 INFO o.a.kafka.common.utils.AppInfoParser akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Kafka startTimeMs: 1691629770967 2023-08-10 01:09:30,971 DEBUG o.a.k.clients.consumer.KafkaConsumer akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [Consumer clientId=consumer-kafkalagexporter-1, groupId=kafkalagexporter] Kafka consumer initialized 2023-08-10 01:09:31,068 INFO o.a.k.c.admin.AdminClientConfig akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - AdminClientConfig values: bootstrap.servers = [b-1.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098, b-2.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098, b-3.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098] client.dns.lookup = use_all_dns_ips client.id = connections.max.idle.ms = 300000 default.api.timeout.ms = 60000 metadata.max.age.ms = 300000 metric.reporters = [] metrics.num.samples = 2 metrics.recording.level = INFO metrics.sample.window.ms = 30000 receive.buffer.bytes = 65536 reconnect.backoff.max.ms = 1000 reconnect.backoff.ms = 50 request.timeout.ms = 10000 retries = 0 retry.backoff.ms = 1000 sasl.client.callback.handler.class = class software.amazon.msk.auth.iam.IAMClientCallbackHandler sasl.jaas.config = [hidden] sasl.kerberos.kinit.cmd = /usr/bin/kinit sasl.kerberos.min.time.before.relogin = 60000 sasl.kerberos.service.name = null sasl.kerberos.ticket.renew.jitter = 0.05 sasl.kerberos.ticket.renew.window.factor = 0.8 sasl.login.callback.handler.class = null sasl.login.class = null sasl.login.connect.timeout.ms = null sasl.login.read.timeout.ms = null sasl.login.refresh.buffer.seconds = 300 sasl.login.refresh.min.period.seconds = 60 sasl.login.refresh.window.factor = 0.8 sasl.login.refresh.window.jitter = 0.05 sasl.login.retry.backoff.max.ms = 10000 sasl.login.retry.backoff.ms = 100 sasl.mechanism = AWS_MSK_IAM sasl.oauthbearer.clock.skew.seconds = 30 sasl.oauthbearer.expected.audience = null sasl.oauthbearer.expected.issuer = null sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000 sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000 sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100 sasl.oauthbearer.jwks.endpoint.url = null sasl.oauthbearer.scope.claim.name = scope sasl.oauthbearer.sub.claim.name = sub sasl.oauthbearer.token.endpoint.url = null security.protocol = SASL_SSL security.providers = null send.buffer.bytes = 131072 socket.connection.setup.timeout.max.ms = 30000 socket.connection.setup.timeout.ms = 10000 ssl.cipher.suites = null ssl.enabled.protocols = [TLSv1.2, TLSv1.3] ssl.endpoint.identification.algorithm = https ssl.engine.factory.class = null ssl.key.password = null ssl.keymanager.algorithm = SunX509 ssl.keystore.certificate.chain = null ssl.keystore.key = null ssl.keystore.location = null ssl.keystore.password = null ssl.keystore.type = JKS ssl.protocol = TLSv1.3 ssl.provider = null ssl.secure.random.implementation = null ssl.trustmanager.algorithm = PKIX ssl.truststore.certificates = null ssl.truststore.location = null ssl.truststore.password = null ssl.truststore.type = JKS 2023-08-10 01:09:31,268 DEBUG o.a.k.c.a.i.AdminMetadataManager akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-1] Setting bootstrap cluster metadata Cluster(id = null, nodes = [b-1.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098 (id: -1 rack: null), b-2.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098 (id: -2 rack: null), b-3.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098 (id: -3 rack: null)], partitions = [], controller = null). 2023-08-10 01:09:31,270 DEBUG s.a.m.a.i.i.MSKCredentialProvider akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Number of options to configure credential provider 0 2023-08-10 01:09:31,272 DEBUG o.a.k.c.s.s.DefaultSslEngineFactory akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Created SSL context with keystore null, truststore null, provider SunJSSE. 2023-08-10 01:09:31,283 INFO o.a.kafka.common.utils.AppInfoParser akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Kafka version: 3.2.3 2023-08-10 01:09:31,283 INFO o.a.kafka.common.utils.AppInfoParser akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Kafka commitId: 50029d3ed8ba576f 2023-08-10 01:09:31,366 INFO o.a.kafka.common.utils.AppInfoParser akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Kafka startTimeMs: 1691629771283 2023-08-10 01:09:31,367 DEBUG o.a.k.clients.admin.KafkaAdminClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-1] Kafka admin client initialized 2023-08-10 01:09:31,473 DEBUG o.a.k.clients.admin.KafkaAdminClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-1] Thread starting 2023-08-10 01:09:31,472 INFO c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Collecting offsets 2023-08-10 01:09:31,480 DEBUG o.apache.kafka.clients.ClientUtils akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Resolved host b-1.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com as 10.0.1.184 2023-08-10 01:09:31,480 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-1] Initiating connection to node b-1.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098 (id: -1 rack: null) using address b-1.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com/10.0.1.184 2023-08-10 01:09:31,479 DEBUG o.a.k.clients.admin.KafkaAdminClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-1] Queueing Call(callName=findAllBrokers, deadlineMs=1691629781478, tries=0, nextAllowedTryMs=0) with a timeout 10000 ms from now. 2023-08-10 01:09:32,779 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-1] Set SASL client state to SEND_APIVERSIONS_REQUEST 2023-08-10 01:09:32,872 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-1] Creating SaslClient: client=null;service=kafka;serviceHostname=b-1.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com;mechs=[AWS_MSK_IAM] 2023-08-10 01:09:32,881 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Setting SASL/AWS_MSK_IAM client state to SEND_CLIENT_FIRST_MESSAGE 2023-08-10 01:09:32,973 DEBUG o.a.kafka.common.network.Selector akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-1] Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -1 2023-08-10 01:09:35,271 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-1] Completed connection to node -1. Fetching API versions. 2023-08-10 01:09:36,285 DEBUG o.a.k.c.network.SslTransportLayer akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [SslTransportLayer channelId=-1 key=channel=java.nio.channels.SocketChannel[connection-pending remote=b-1.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com/10.0.1.184:9098], selector=sun.nio.ch.EPollSelectorImpl@718739c, interestOps=8, readyOps=0] SSL handshake completed successfully with peerHost 'b-1.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com' peerPort 9098 peerPrincipal 'CN=*.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com' cipherSuite 'TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384' 2023-08-10 01:09:37,070 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-1] Set SASL client state to RECEIVE_APIVERSIONS_RESPONSE 2023-08-10 01:09:37,275 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-1] Set SASL client state to SEND_HANDSHAKE_REQUEST 2023-08-10 01:09:37,276 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-1] Set SASL client state to RECEIVE_HANDSHAKE_RESPONSE 2023-08-10 01:09:37,277 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-1] Set SASL client state to INITIAL 2023-08-10 01:09:37,278 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - State SEND_CLIENT_FIRST_MESSAGE at start of evaluating challenge 2023-08-10 01:09:37,280 DEBUG s.a.m.a.iam.IAMClientCallbackHandler akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Type information for callback: class: software.amazon.msk.auth.iam.internals.AWSCredentialsCallback classloader: jdk.internal.loader.ClassLoaders$AppClassLoader@53ba4f8c from class: software.amazon.msk.auth.iam.IAMClientCallbackHandler classloader: jdk.internal.loader.ClassLoaders$AppClassLoader@53ba4f8c 2023-08-10 01:09:37,280 DEBUG s.a.m.a.iam.IAMClientCallbackHandler akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Selecting provider software.amazon.msk.auth.iam.internals.MSKCredentialProvider to load credentials 2023-08-10 01:09:41,676 DEBUG s.a.m.a.i.i.AWS4SignedPayloadGenerator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Signer configured for kafka-cluster service and ap-southeast-2 region 2023-08-10 01:09:42,069 DEBUG o.a.k.clients.admin.KafkaAdminClient - [AdminClient clientId=adminclient-1] Initiating close operation. 2023-08-10 01:09:42,069 DEBUG o.a.k.clients.admin.KafkaAdminClient - [AdminClient clientId=adminclient-1] Waiting for the I/O thread to exit. Hard shutdown in 10000 ms. 2023-08-10 01:09:42,085 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Setting SASL/AWS_MSK_IAM client state to RECEIVE_SERVER_RESPONSE 2023-08-10 01:09:42,085 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - State RECEIVE_SERVER_RESPONSE at end of evaluating challenge 2023-08-10 01:09:42,171 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-1] Set SASL client state to INTERMEDIATE 2023-08-10 01:09:42,174 INFO o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-1] Disconnecting from node -1 due to socket connection setup timeout. The timeout value is 10690 ms. 2023-08-10 01:09:42,178 DEBUG o.a.k.clients.admin.KafkaAdminClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-1] Hard shutdown in 9892 ms. 2023-08-10 01:09:42,266 DEBUG o.a.k.clients.admin.KafkaAdminClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-1] Call(callName=findAllBrokers, deadlineMs=1691629781478, tries=1, nextAllowedTryMs=1691629783177) timed out at 1691629782177 after 1 attempt(s) java.lang.Exception: TimeoutException: Timed out waiting for a node assignment. Call: findAllBrokers at org.apache.kafka.clients.admin.KafkaAdminClient$Call.handleTimeoutFailure(KafkaAdminClient.java:843) at org.apache.kafka.clients.admin.KafkaAdminClient$Call.fail(KafkaAdminClient.java:812) at org.apache.kafka.clients.admin.KafkaAdminClient$TimeoutProcessor.handleTimeouts(KafkaAdminClient.java:942) at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.timeoutPendingCalls(KafkaAdminClient.java:1021) at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1375) at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1339) at java.base/java.lang.Thread.run(Unknown Source) 2023-08-10 01:09:42,269 DEBUG o.a.k.clients.admin.KafkaAdminClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-1] Call(callName=fetchMetadata, deadlineMs=1691629781473, tries=1, nextAllowedTryMs=1691629783177) timed out at 1691629782177 after 1 attempt(s) java.lang.Exception: TimeoutException: Timed out waiting for a node assignment. Call: fetchMetadata at org.apache.kafka.clients.admin.KafkaAdminClient$Call.handleTimeoutFailure(KafkaAdminClient.java:843) at org.apache.kafka.clients.admin.KafkaAdminClient$Call.fail(KafkaAdminClient.java:812) at org.apache.kafka.clients.admin.KafkaAdminClient$TimeoutProcessor.handleTimeouts(KafkaAdminClient.java:942) at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.timeoutPendingCalls(KafkaAdminClient.java:1021) at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1375) at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1339) at java.base/java.lang.Thread.run(Unknown Source) 2023-08-10 01:09:42,270 INFO o.a.k.c.a.i.AdminMetadataManager akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-1] Metadata update failed org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: fetchMetadata 2023-08-10 01:09:42,270 DEBUG o.a.k.clients.admin.KafkaAdminClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-1] Timed out 2 pending calls. 2023-08-10 01:09:42,270 DEBUG o.apache.kafka.clients.ClientUtils akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Resolved host b-3.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com as 10.0.5.125 2023-08-10 01:09:42,271 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-1] Initiating connection to node b-3.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098 (id: -3 rack: null) using address b-3.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com/10.0.5.125 2023-08-10 01:09:42,271 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-1] Set SASL client state to SEND_APIVERSIONS_REQUEST 2023-08-10 01:09:42,271 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-1] Creating SaslClient: client=null;service=kafka;serviceHostname=b-3.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com;mechs=[AWS_MSK_IAM] 2023-08-10 01:09:42,272 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Setting SASL/AWS_MSK_IAM client state to SEND_CLIENT_FIRST_MESSAGE 2023-08-10 01:09:42,274 DEBUG o.a.kafka.common.network.Selector akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-1] Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -3 2023-08-10 01:09:42,283 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-1] Completed connection to node -3. Fetching API versions. 2023-08-10 01:09:42,367 INFO o.a.kafka.common.utils.AppInfoParser akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - App info kafka.admin.client for adminclient-1 unregistered 2023-08-10 01:09:42,367 INFO o.a.k.c.a.i.AdminMetadataManager akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-1] Metadata update failed org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: fetchMetadata 2023-08-10 01:09:42,367 DEBUG o.a.k.clients.admin.KafkaAdminClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-1] Timed out 1 call(s) with assigned nodes. 2023-08-10 01:09:42,368 INFO o.a.k.clients.admin.KafkaAdminClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-1] Timed out 1 remaining operation(s) during close. 2023-08-10 01:09:42,369 DEBUG o.a.k.c.network.SslTransportLayer akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [SslTransportLayer channelId=-3 key=channel=java.nio.channels.SocketChannel[connection-pending remote=b-3.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com/10.0.5.125:9098], selector=sun.nio.ch.EPollSelectorImpl@718739c, interestOps=8, readyOps=0] Failed to send SSL Close message java.io.IOException: Unexpected status returned by SSLEngine.wrap, expected CLOSED, received OK. Will not send close message to peer. at org.apache.kafka.common.network.SslTransportLayer.close(SslTransportLayer.java:194) at org.apache.kafka.common.utils.Utils.closeAll(Utils.java:975) at org.apache.kafka.common.network.KafkaChannel.close(KafkaChannel.java:155) at org.apache.kafka.common.network.Selector.doClose(Selector.java:955) at org.apache.kafka.common.network.Selector.close(Selector.java:939) at org.apache.kafka.common.network.Selector.close(Selector.java:885) at org.apache.kafka.common.network.Selector.lambda$null$0(Selector.java:368) at org.apache.kafka.common.utils.Utils.closeQuietly(Utils.java:1015) at org.apache.kafka.common.utils.Utils.closeAllQuietly(Utils.java:1030) at org.apache.kafka.common.network.Selector.close(Selector.java:367) at org.apache.kafka.clients.NetworkClient.close(NetworkClient.java:653) at org.apache.kafka.common.utils.Utils.closeQuietly(Utils.java:1005) at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1356) at java.base/java.lang.Thread.run(Unknown Source) 2023-08-10 01:09:42,376 INFO o.a.kafka.common.metrics.Metrics akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Metrics scheduler closed 2023-08-10 01:09:42,377 INFO o.a.kafka.common.metrics.Metrics akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Closing reporter org.apache.kafka.common.metrics.JmxReporter 2023-08-10 01:09:42,377 INFO o.a.kafka.common.metrics.Metrics akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Metrics reporters closed 2023-08-10 01:09:42,377 DEBUG o.a.k.clients.admin.KafkaAdminClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-1] Exiting AdminClientRunnable thread. 2023-08-10 01:09:42,466 DEBUG o.a.k.clients.admin.KafkaAdminClient - [AdminClient clientId=adminclient-1] Kafka admin client closed. 2023-08-10 01:09:42,467 DEBUG o.a.k.c.c.i.ConsumerCoordinator - [Consumer clientId=consumer-kafkalagexporter-1, groupId=kafkalagexporter] Executing onLeavePrepare with generation Generation{generationId=-1, memberId='', protocol='null'} 2023-08-10 01:09:42,467 INFO o.a.k.c.c.i.ConsumerCoordinator - [Consumer clientId=consumer-kafkalagexporter-1, groupId=kafkalagexporter] Resetting generation and member id due to: consumer pro-actively leaving the group 2023-08-10 01:09:42,467 INFO o.a.k.c.c.i.ConsumerCoordinator - [Consumer clientId=consumer-kafkalagexporter-1, groupId=kafkalagexporter] Request joining group due to: consumer pro-actively leaving the group 2023-08-10 01:09:42,469 INFO o.a.kafka.common.metrics.Metrics - Metrics scheduler closed 2023-08-10 01:09:42,469 INFO o.a.kafka.common.metrics.Metrics - Closing reporter org.apache.kafka.common.metrics.JmxReporter 2023-08-10 01:09:42,470 INFO o.a.kafka.common.metrics.Metrics - Metrics reporters closed 2023-08-10 01:09:42,478 INFO o.a.kafka.common.utils.AppInfoParser - App info kafka.consumer for consumer-kafkalagexporter-1 unregistered 2023-08-10 01:09:42,478 DEBUG o.a.k.clients.consumer.KafkaConsumer - [Consumer clientId=consumer-kafkalagexporter-1, groupId=kafkalagexporter] Kafka consumer has been closed 2023-08-10 01:09:42,479 INFO c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Clearing all metrics before shutdown 2023-08-10 01:09:42,489 ERROR c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Supervisor RestartSupervisor saw failure [1]: A failure occurred while retrieving offsets. Shutting down. java.lang.Exception: A failure occurred while retrieving offsets. Shutting down. at com.lightbend.kafkalagexporter.ConsumerGroupCollector$CollectorBehavior.$anonfun$collector$1(ConsumerGroupCollector.scala:302) at akka.actor.typed.internal.BehaviorImpl$ReceiveBehavior.receive(BehaviorImpl.scala:136) at akka.actor.typed.Behavior$.interpret(Behavior.scala:274) at akka.actor.typed.Behavior$.interpretMessage(Behavior.scala:230) at akka.actor.typed.internal.InterceptorImpl$$anon$2.apply(InterceptorImpl.scala:57) at akka.actor.typed.internal.RestartSupervisor.aroundReceive(Supervision.scala:279) at akka.actor.typed.internal.InterceptorImpl.receive(InterceptorImpl.scala:85) at akka.actor.typed.Behavior$.interpret(Behavior.scala:274) at akka.actor.typed.Behavior$.interpretMessage(Behavior.scala:230) at akka.actor.typed.internal.adapter.ActorAdapter.handleMessage(ActorAdapter.scala:131) at akka.actor.typed.internal.adapter.ActorAdapter.aroundReceive(ActorAdapter.scala:107) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:579) at akka.actor.ActorCell.invoke(ActorCell.scala:547) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270) at akka.dispatch.Mailbox.run(Mailbox.scala:231) at akka.dispatch.Mailbox.exec(Mailbox.scala:243) at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source) at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source) at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source) Caused by: java.util.concurrent.TimeoutException: null at java.base/java.util.concurrent.CompletableFuture.timedGet(Unknown Source) at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source) at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:180) at com.lightbend.kafkalagexporter.KafkaClient$.$anonfun$kafkaFuture$1(KafkaClient.scala:85) at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659) at scala.util.Success.$anonfun$map$1(Try.scala:255) at scala.util.Success.map(Try.scala:213) at scala.concurrent.Future.$anonfun$map$1(Future.scala:292) at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33) at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) 2023-08-10 01:09:43,765 INFO c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Spawned ConsumerGroupCollector for cluster: kafka-messaging 2023-08-10 01:09:43,766 INFO o.a.k.c.consumer.ConsumerConfig akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - ConsumerConfig values: allow.auto.create.topics = true auto.commit.interval.ms = 5000 auto.offset.reset = latest bootstrap.servers = [b-1.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098, b-2.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098, b-3.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098] check.crcs = true client.dns.lookup = use_all_dns_ips client.id = consumer-kafkalagexporter-2 client.rack = connections.max.idle.ms = 540000 default.api.timeout.ms = 60000 enable.auto.commit = false exclude.internal.topics = true fetch.max.bytes = 52428800 fetch.max.wait.ms = 500 fetch.min.bytes = 1 group.id = kafkalagexporter group.instance.id = null heartbeat.interval.ms = 3000 interceptor.classes = [] internal.leave.group.on.close = true internal.throw.on.fetch.stable.offset.unsupported = false isolation.level = read_uncommitted key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer max.partition.fetch.bytes = 1048576 max.poll.interval.ms = 300000 max.poll.records = 500 metadata.max.age.ms = 300000 metric.reporters = [] metrics.num.samples = 2 metrics.recording.level = INFO metrics.sample.window.ms = 30000 partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor, class org.apache.kafka.clients.consumer.CooperativeStickyAssignor] receive.buffer.bytes = 65536 reconnect.backoff.max.ms = 1000 reconnect.backoff.ms = 50 request.timeout.ms = 10000 retry.backoff.ms = 1000 sasl.client.callback.handler.class = class software.amazon.msk.auth.iam.IAMClientCallbackHandler sasl.jaas.config = [hidden] sasl.kerberos.kinit.cmd = /usr/bin/kinit sasl.kerberos.min.time.before.relogin = 60000 sasl.kerberos.service.name = null sasl.kerberos.ticket.renew.jitter = 0.05 sasl.kerberos.ticket.renew.window.factor = 0.8 sasl.login.callback.handler.class = null sasl.login.class = null sasl.login.connect.timeout.ms = null sasl.login.read.timeout.ms = null sasl.login.refresh.buffer.seconds = 300 sasl.login.refresh.min.period.seconds = 60 sasl.login.refresh.window.factor = 0.8 sasl.login.refresh.window.jitter = 0.05 sasl.login.retry.backoff.max.ms = 10000 sasl.login.retry.backoff.ms = 100 sasl.mechanism = AWS_MSK_IAM sasl.oauthbearer.clock.skew.seconds = 30 sasl.oauthbearer.expected.audience = null sasl.oauthbearer.expected.issuer = null sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000 sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000 sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100 sasl.oauthbearer.jwks.endpoint.url = null sasl.oauthbearer.scope.claim.name = scope sasl.oauthbearer.sub.claim.name = sub sasl.oauthbearer.token.endpoint.url = null security.protocol = SASL_SSL security.providers = null send.buffer.bytes = 131072 session.timeout.ms = 45000 socket.connection.setup.timeout.max.ms = 30000 socket.connection.setup.timeout.ms = 10000 ssl.cipher.suites = null ssl.enabled.protocols = [TLSv1.2, TLSv1.3] ssl.endpoint.identification.algorithm = https ssl.engine.factory.class = null ssl.key.password = null ssl.keymanager.algorithm = SunX509 ssl.keystore.certificate.chain = null ssl.keystore.key = null ssl.keystore.location = null ssl.keystore.password = null ssl.keystore.type = JKS ssl.protocol = TLSv1.3 ssl.provider = null ssl.secure.random.implementation = null ssl.trustmanager.algorithm = PKIX ssl.truststore.certificates = null ssl.truststore.location = null ssl.truststore.password = null ssl.truststore.type = JKS value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer 2023-08-10 01:09:43,767 DEBUG o.a.k.clients.consumer.KafkaConsumer akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [Consumer clientId=consumer-kafkalagexporter-2, groupId=kafkalagexporter] Initializing the Kafka consumer 2023-08-10 01:09:43,769 DEBUG s.a.m.a.i.i.MSKCredentialProvider akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Number of options to configure credential provider 0 2023-08-10 01:09:43,770 DEBUG s.amazon.msk.auth.iam.IAMLoginModule akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - IAMLoginModule initialized 2023-08-10 01:09:43,770 INFO o.a.k.c.s.a.AbstractLogin akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Successfully logged in. 2023-08-10 01:09:43,770 DEBUG o.a.k.c.s.s.DefaultSslEngineFactory akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Created SSL context with keystore null, truststore null, provider SunJSSE. 2023-08-10 01:09:43,778 INFO o.a.kafka.common.utils.AppInfoParser akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Kafka version: 3.2.3 2023-08-10 01:09:43,778 INFO o.a.kafka.common.utils.AppInfoParser akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Kafka commitId: 50029d3ed8ba576f 2023-08-10 01:09:43,778 INFO o.a.kafka.common.utils.AppInfoParser akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Kafka startTimeMs: 1691629783778 2023-08-10 01:09:43,779 DEBUG o.a.k.clients.consumer.KafkaConsumer akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [Consumer clientId=consumer-kafkalagexporter-2, groupId=kafkalagexporter] Kafka consumer initialized 2023-08-10 01:09:43,779 INFO o.a.k.c.admin.AdminClientConfig akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - AdminClientConfig values: bootstrap.servers = [b-1.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098, b-2.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098, b-3.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098] client.dns.lookup = use_all_dns_ips client.id = connections.max.idle.ms = 300000 default.api.timeout.ms = 60000 metadata.max.age.ms = 300000 metric.reporters = [] metrics.num.samples = 2 metrics.recording.level = INFO metrics.sample.window.ms = 30000 receive.buffer.bytes = 65536 reconnect.backoff.max.ms = 1000 reconnect.backoff.ms = 50 request.timeout.ms = 10000 retries = 0 retry.backoff.ms = 1000 sasl.client.callback.handler.class = class software.amazon.msk.auth.iam.IAMClientCallbackHandler sasl.jaas.config = [hidden] sasl.kerberos.kinit.cmd = /usr/bin/kinit sasl.kerberos.min.time.before.relogin = 60000 sasl.kerberos.service.name = null sasl.kerberos.ticket.renew.jitter = 0.05 sasl.kerberos.ticket.renew.window.factor = 0.8 sasl.login.callback.handler.class = null sasl.login.class = null sasl.login.connect.timeout.ms = null sasl.login.read.timeout.ms = null sasl.login.refresh.buffer.seconds = 300 sasl.login.refresh.min.period.seconds = 60 sasl.login.refresh.window.factor = 0.8 sasl.login.refresh.window.jitter = 0.05 sasl.login.retry.backoff.max.ms = 10000 sasl.login.retry.backoff.ms = 100 sasl.mechanism = AWS_MSK_IAM sasl.oauthbearer.clock.skew.seconds = 30 sasl.oauthbearer.expected.audience = null sasl.oauthbearer.expected.issuer = null sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000 sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000 sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100 sasl.oauthbearer.jwks.endpoint.url = null sasl.oauthbearer.scope.claim.name = scope sasl.oauthbearer.sub.claim.name = sub sasl.oauthbearer.token.endpoint.url = null security.protocol = SASL_SSL security.providers = null send.buffer.bytes = 131072 socket.connection.setup.timeout.max.ms = 30000 socket.connection.setup.timeout.ms = 10000 ssl.cipher.suites = null ssl.enabled.protocols = [TLSv1.2, TLSv1.3] ssl.endpoint.identification.algorithm = https ssl.engine.factory.class = null ssl.key.password = null ssl.keymanager.algorithm = SunX509 ssl.keystore.certificate.chain = null ssl.keystore.key = null ssl.keystore.location = null ssl.keystore.password = null ssl.keystore.type = JKS ssl.protocol = TLSv1.3 ssl.provider = null ssl.secure.random.implementation = null ssl.trustmanager.algorithm = PKIX ssl.truststore.certificates = null ssl.truststore.location = null ssl.truststore.password = null ssl.truststore.type = JKS 2023-08-10 01:09:43,780 DEBUG o.a.k.c.a.i.AdminMetadataManager akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Setting bootstrap cluster metadata Cluster(id = null, nodes = [b-2.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098 (id: -2 rack: null), b-3.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098 (id: -3 rack: null), b-1.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098 (id: -1 rack: null)], partitions = [], controller = null). 2023-08-10 01:09:43,780 DEBUG s.a.m.a.i.i.MSKCredentialProvider akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Number of options to configure credential provider 0 2023-08-10 01:09:43,781 DEBUG o.a.k.c.s.s.DefaultSslEngineFactory akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Created SSL context with keystore null, truststore null, provider SunJSSE. 2023-08-10 01:09:43,867 INFO o.a.kafka.common.utils.AppInfoParser akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Kafka version: 3.2.3 2023-08-10 01:09:43,867 INFO o.a.kafka.common.utils.AppInfoParser akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Kafka commitId: 50029d3ed8ba576f 2023-08-10 01:09:43,867 INFO o.a.kafka.common.utils.AppInfoParser akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Kafka startTimeMs: 1691629783867 2023-08-10 01:09:43,868 DEBUG o.a.k.clients.admin.KafkaAdminClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Kafka admin client initialized 2023-08-10 01:09:43,868 INFO c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Collecting offsets 2023-08-10 01:09:43,869 DEBUG o.a.k.clients.admin.KafkaAdminClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Queueing Call(callName=findAllBrokers, deadlineMs=1691629793868, tries=0, nextAllowedTryMs=0) with a timeout 10000 ms from now. 2023-08-10 01:09:43,869 DEBUG o.a.k.clients.admin.KafkaAdminClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Thread starting 2023-08-10 01:09:43,870 DEBUG o.a.k.c.a.i.AdminMetadataManager akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Requesting metadata update. 2023-08-10 01:09:43,870 DEBUG o.apache.kafka.clients.ClientUtils akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Resolved host b-2.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com as 10.0.3.55 2023-08-10 01:09:43,870 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Initiating connection to node b-2.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098 (id: -2 rack: null) using address b-2.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com/10.0.3.55 2023-08-10 01:09:43,871 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Set SASL client state to SEND_APIVERSIONS_REQUEST 2023-08-10 01:09:43,871 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Creating SaslClient: client=null;service=kafka;serviceHostname=b-2.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com;mechs=[AWS_MSK_IAM] 2023-08-10 01:09:43,872 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Setting SASL/AWS_MSK_IAM client state to SEND_CLIENT_FIRST_MESSAGE 2023-08-10 01:09:43,873 DEBUG o.a.kafka.common.network.Selector akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -2 2023-08-10 01:09:43,974 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Completed connection to node -2. Fetching API versions. 2023-08-10 01:09:44,074 DEBUG o.a.k.c.network.SslTransportLayer akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [SslTransportLayer channelId=-2 key=channel=java.nio.channels.SocketChannel[connection-pending remote=b-2.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com/10.0.3.55:9098], selector=sun.nio.ch.EPollSelectorImpl@4690ee58, interestOps=8, readyOps=0] SSL handshake completed successfully with peerHost 'b-2.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com' peerPort 9098 peerPrincipal 'CN=*.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com' cipherSuite 'TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384' 2023-08-10 01:09:44,077 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Set SASL client state to RECEIVE_APIVERSIONS_RESPONSE 2023-08-10 01:09:44,081 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Set SASL client state to SEND_HANDSHAKE_REQUEST 2023-08-10 01:09:44,081 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Set SASL client state to RECEIVE_HANDSHAKE_RESPONSE 2023-08-10 01:09:44,167 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Set SASL client state to INITIAL 2023-08-10 01:09:44,168 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - State SEND_CLIENT_FIRST_MESSAGE at start of evaluating challenge 2023-08-10 01:09:44,168 DEBUG s.a.m.a.iam.IAMClientCallbackHandler akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Type information for callback: class: software.amazon.msk.auth.iam.internals.AWSCredentialsCallback classloader: jdk.internal.loader.ClassLoaders$AppClassLoader@53ba4f8c from class: software.amazon.msk.auth.iam.IAMClientCallbackHandler classloader: jdk.internal.loader.ClassLoaders$AppClassLoader@53ba4f8c 2023-08-10 01:09:44,168 DEBUG s.a.m.a.iam.IAMClientCallbackHandler akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Selecting provider software.amazon.msk.auth.iam.internals.MSKCredentialProvider to load credentials 2023-08-10 01:09:44,172 DEBUG s.a.m.a.i.i.AWS4SignedPayloadGenerator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Signer configured for kafka-cluster service and ap-southeast-2 region 2023-08-10 01:09:44,269 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Setting SASL/AWS_MSK_IAM client state to RECEIVE_SERVER_RESPONSE 2023-08-10 01:09:44,269 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - State RECEIVE_SERVER_RESPONSE at end of evaluating challenge 2023-08-10 01:09:44,272 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Set SASL client state to INTERMEDIATE 2023-08-10 01:09:44,298 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - State RECEIVE_SERVER_RESPONSE at start of evaluating challenge 2023-08-10 01:09:44,376 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Response from server: AuthenticationResponse(version=2020_10_22, requestId=6bce71b3-970f-4021-8711-9b9d906ce776) 2023-08-10 01:09:44,376 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Setting SASL/AWS_MSK_IAM client state to COMPLETE 2023-08-10 01:09:44,376 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - State COMPLETE at end of evaluating challenge 2023-08-10 01:09:44,376 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Set SASL client state to COMPLETE 2023-08-10 01:09:44,376 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Finished authentication with session expiration in 21553703 ms and session re-authentication on or after 20420624 ms 2023-08-10 01:09:44,377 DEBUG o.a.kafka.common.network.Selector akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Successfully authenticated with b-2.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com/10.0.3.55 2023-08-10 01:09:44,377 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Initiating API versions fetch from node -2. 2023-08-10 01:09:44,378 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Sending API_VERSIONS request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=adminclient-2, correlationId=0) and timeout 3600000 to node -2: ApiVersionsRequestData(clientSoftwareName='apache-kafka-java', clientSoftwareVersion='3.2.3') 2023-08-10 01:09:44,472 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Received API_VERSIONS response from node -2 for request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=adminclient-2, correlationId=0): ApiVersionsResponseData(errorCode=0, apiKeys=[ApiVersion(apiKey=0, minVersion=0, maxVersion=9), ApiVersion(apiKey=1, minVersion=0, maxVersion=13), ApiVersion(apiKey=2, minVersion=0, maxVersion=7), ApiVersion(apiKey=3, minVersion=0, maxVersion=12), ApiVersion(apiKey=4, minVersion=0, maxVersion=6), ApiVersion(apiKey=5, minVersion=0, maxVersion=3), ApiVersion(apiKey=6, minVersion=0, maxVersion=7), ApiVersion(apiKey=7, minVersion=0, maxVersion=3), ApiVersion(apiKey=8, minVersion=0, maxVersion=8), ApiVersion(apiKey=9, minVersion=0, maxVersion=8), ApiVersion(apiKey=10, minVersion=0, maxVersion=4), ApiVersion(apiKey=11, minVersion=0, maxVersion=9), ApiVersion(apiKey=12, minVersion=0, maxVersion=4), ApiVersion(apiKey=13, minVersion=0, maxVersion=5), ApiVersion(apiKey=14, minVersion=0, maxVersion=5), ApiVersion(apiKey=15, minVersion=0, maxVersion=5), ApiVersion(apiKey=16, minVersion=0, maxVersion=4), ApiVersion(apiKey=17, minVersion=0, maxVersion=1), ApiVersion(apiKey=18, minVersion=0, maxVersion=3), ApiVersion(apiKey=19, minVersion=0, maxVersion=7), ApiVersion(apiKey=20, minVersion=0, maxVersion=6), ApiVersion(apiKey=21, minVersion=0, maxVersion=2), ApiVersion(apiKey=22, minVersion=0, maxVersion=4), ApiVersion(apiKey=23, minVersion=0, maxVersion=4), ApiVersion(apiKey=24, minVersion=0, maxVersion=3), ApiVersion(apiKey=25, minVersion=0, maxVersion=3), ApiVersion(apiKey=26, minVersion=0, maxVersion=3), ApiVersion(apiKey=27, minVersion=0, maxVersion=1), ApiVersion(apiKey=28, minVersion=0, maxVersion=3), ApiVersion(apiKey=29, minVersion=0, maxVersion=3), ApiVersion(apiKey=30, minVersion=0, maxVersion=3), ApiVersion(apiKey=31, minVersion=0, maxVersion=3), ApiVersion(apiKey=32, minVersion=0, maxVersion=4), ApiVersion(apiKey=33, minVersion=0, maxVersion=2), ApiVersion(apiKey=34, minVersion=0, maxVersion=2), ApiVersion(apiKey=35, minVersion=0, maxVersion=4), ApiVersion(apiKey=36, minVersion=0, maxVersion=2), ApiVersion(apiKey=37, minVersion=0, maxVersion=3), ApiVersion(apiKey=38, minVersion=0, maxVersion=3), ApiVersion(apiKey=39, minVersion=0, maxVersion=2), ApiVersion(apiKey=40, minVersion=0, maxVersion=2), ApiVersion(apiKey=41, minVersion=0, maxVersion=3), ApiVersion(apiKey=42, minVersion=0, maxVersion=2), ApiVersion(apiKey=43, minVersion=0, maxVersion=2), ApiVersion(apiKey=44, minVersion=0, maxVersion=1), ApiVersion(apiKey=45, minVersion=0, maxVersion=0), ApiVersion(apiKey=46, minVersion=0, maxVersion=0), ApiVersion(apiKey=47, minVersion=0, maxVersion=0), ApiVersion(apiKey=48, minVersion=0, maxVersion=1), ApiVersion(apiKey=49, minVersion=0, maxVersion=1), ApiVersion(apiKey=50, minVersion=0, maxVersion=0), ApiVersion(apiKey=51, minVersion=0, maxVersion=0), ApiVersion(apiKey=56, minVersion=0, maxVersion=2), ApiVersion(apiKey=57, minVersion=0, maxVersion=1), ApiVersion(apiKey=60, minVersion=0, maxVersion=0), ApiVersion(apiKey=61, minVersion=0, maxVersion=0), ApiVersion(apiKey=65, minVersion=0, maxVersion=0), ApiVersion(apiKey=66, minVersion=0, maxVersion=0), ApiVersion(apiKey=67, minVersion=0, maxVersion=0)], throttleTimeMs=0, supportedFeatures=[], finalizedFeaturesEpoch=0, finalizedFeatures=[]) 2023-08-10 01:09:44,477 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Node -2 has finalized features epoch: 0, finalized features: [], supported features: [], API versions: (Produce(0): 0 to 9 [usable: 9], Fetch(1): 0 to 13 [usable: 13], ListOffsets(2): 0 to 7 [usable: 7], Metadata(3): 0 to 12 [usable: 12], LeaderAndIsr(4): 0 to 6 [usable: 6], StopReplica(5): 0 to 3 [usable: 3], UpdateMetadata(6): 0 to 7 [usable: 7], ControlledShutdown(7): 0 to 3 [usable: 3], OffsetCommit(8): 0 to 8 [usable: 8], OffsetFetch(9): 0 to 8 [usable: 8], FindCoordinator(10): 0 to 4 [usable: 4], JoinGroup(11): 0 to 9 [usable: 9], Heartbeat(12): 0 to 4 [usable: 4], LeaveGroup(13): 0 to 5 [usable: 5], SyncGroup(14): 0 to 5 [usable: 5], DescribeGroups(15): 0 to 5 [usable: 5], ListGroups(16): 0 to 4 [usable: 4], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 3 [usable: 3], CreateTopics(19): 0 to 7 [usable: 7], DeleteTopics(20): 0 to 6 [usable: 6], DeleteRecords(21): 0 to 2 [usable: 2], InitProducerId(22): 0 to 4 [usable: 4], OffsetForLeaderEpoch(23): 0 to 4 [usable: 4], AddPartitionsToTxn(24): 0 to 3 [usable: 3], AddOffsetsToTxn(25): 0 to 3 [usable: 3], EndTxn(26): 0 to 3 [usable: 3], WriteTxnMarkers(27): 0 to 1 [usable: 1], TxnOffsetCommit(28): 0 to 3 [usable: 3], DescribeAcls(29): 0 to 3 [usable: 2], CreateAcls(30): 0 to 3 [usable: 2], DeleteAcls(31): 0 to 3 [usable: 2], DescribeConfigs(32): 0 to 4 [usable: 4], AlterConfigs(33): 0 to 2 [usable: 2], AlterReplicaLogDirs(34): 0 to 2 [usable: 2], DescribeLogDirs(35): 0 to 4 [usable: 3], SaslAuthenticate(36): 0 to 2 [usable: 2], CreatePartitions(37): 0 to 3 [usable: 3], CreateDelegationToken(38): 0 to 3 [usable: 2], RenewDelegationToken(39): 0 to 2 [usable: 2], ExpireDelegationToken(40): 0 to 2 [usable: 2], DescribeDelegationToken(41): 0 to 3 [usable: 2], DeleteGroups(42): 0 to 2 [usable: 2], ElectLeaders(43): 0 to 2 [usable: 2], IncrementalAlterConfigs(44): 0 to 1 [usable: 1], AlterPartitionReassignments(45): 0 [usable: 0], ListPartitionReassignments(46): 0 [usable: 0], OffsetDelete(47): 0 [usable: 0], DescribeClientQuotas(48): 0 to 1 [usable: 1], AlterClientQuotas(49): 0 to 1 [usable: 1], DescribeUserScramCredentials(50): 0 [usable: 0], AlterUserScramCredentials(51): 0 [usable: 0], AlterPartition(56): 0 to 2 [usable: 1], UpdateFeatures(57): 0 to 1 [usable: 0], DescribeCluster(60): 0 [usable: 0], DescribeProducers(61): 0 [usable: 0], DescribeTransactions(65): 0 [usable: 0], ListTransactions(66): 0 [usable: 0], AllocateProducerIds(67): 0 [usable: 0]). 2023-08-10 01:09:44,478 DEBUG o.a.k.clients.admin.KafkaAdminClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Sending MetadataRequestData(topics=[], allowAutoTopicCreation=true, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false) to b-2.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098 (id: -2 rack: null). correlationId=1, timeoutMs=9393 2023-08-10 01:09:44,482 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Sending METADATA request with header RequestHeader(apiKey=METADATA, apiVersion=12, clientId=adminclient-2, correlationId=1) and timeout 9393 to node -2: MetadataRequestData(topics=[], allowAutoTopicCreation=true, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false) 2023-08-10 01:09:44,485 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Received METADATA response from node -2 for request with header RequestHeader(apiKey=METADATA, apiVersion=12, clientId=adminclient-2, correlationId=1): MetadataResponseData(throttleTimeMs=0, brokers=[MetadataResponseBroker(nodeId=2, host='b-2.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com', port=9098, rack='apse2-az3'), MetadataResponseBroker(nodeId=3, host='b-3.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com', port=9098, rack='apse2-az2'), MetadataResponseBroker(nodeId=1, host='b-1.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com', port=9098, rack='apse2-az1')], clusterId='E7DxCekgTJKvUyP3nH8P2w', controllerId=1, topics=[], clusterAuthorizedOperations=-2147483648) 2023-08-10 01:09:44,571 DEBUG o.a.k.c.a.i.AdminMetadataManager akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Updating cluster metadata to Cluster(id = E7DxCekgTJKvUyP3nH8P2w, nodes = [b-1.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098 (id: 1 rack: apse2-az1), b-2.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098 (id: 2 rack: apse2-az3), b-3.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098 (id: 3 rack: apse2-az2)], partitions = [], controller = b-1.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098 (id: 1 rack: apse2-az1)) 2023-08-10 01:09:44,572 DEBUG o.apache.kafka.clients.ClientUtils akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Resolved host b-3.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com as 10.0.5.125 2023-08-10 01:09:44,572 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Initiating connection to node b-3.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098 (id: 3 rack: apse2-az2) using address b-3.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com/10.0.5.125 2023-08-10 01:09:44,573 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Set SASL client state to SEND_APIVERSIONS_REQUEST 2023-08-10 01:09:44,573 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Creating SaslClient: client=null;service=kafka;serviceHostname=b-3.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com;mechs=[AWS_MSK_IAM] 2023-08-10 01:09:44,574 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Setting SASL/AWS_MSK_IAM client state to SEND_CLIENT_FIRST_MESSAGE 2023-08-10 01:09:44,576 DEBUG o.a.kafka.common.network.Selector akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node 3 2023-08-10 01:09:44,669 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Completed connection to node 3. Fetching API versions. 2023-08-10 01:09:44,770 DEBUG o.a.k.c.network.SslTransportLayer akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [SslTransportLayer channelId=3 key=channel=java.nio.channels.SocketChannel[connection-pending remote=b-3.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com/10.0.5.125:9098], selector=sun.nio.ch.EPollSelectorImpl@4690ee58, interestOps=8, readyOps=0] SSL handshake completed successfully with peerHost 'b-3.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com' peerPort 9098 peerPrincipal 'CN=*.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com' cipherSuite 'TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384' 2023-08-10 01:09:44,773 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Set SASL client state to RECEIVE_APIVERSIONS_RESPONSE 2023-08-10 01:09:44,775 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Set SASL client state to SEND_HANDSHAKE_REQUEST 2023-08-10 01:09:44,779 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Set SASL client state to RECEIVE_HANDSHAKE_RESPONSE 2023-08-10 01:09:44,781 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Set SASL client state to INITIAL 2023-08-10 01:09:44,781 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - State SEND_CLIENT_FIRST_MESSAGE at start of evaluating challenge 2023-08-10 01:09:44,781 DEBUG s.a.m.a.iam.IAMClientCallbackHandler akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Type information for callback: class: software.amazon.msk.auth.iam.internals.AWSCredentialsCallback classloader: jdk.internal.loader.ClassLoaders$AppClassLoader@53ba4f8c from class: software.amazon.msk.auth.iam.IAMClientCallbackHandler classloader: jdk.internal.loader.ClassLoaders$AppClassLoader@53ba4f8c 2023-08-10 01:09:44,866 DEBUG s.a.m.a.iam.IAMClientCallbackHandler akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Selecting provider software.amazon.msk.auth.iam.internals.MSKCredentialProvider to load credentials 2023-08-10 01:09:44,868 DEBUG s.a.m.a.i.i.AWS4SignedPayloadGenerator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Signer configured for kafka-cluster service and ap-southeast-2 region 2023-08-10 01:09:44,873 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Setting SASL/AWS_MSK_IAM client state to RECEIVE_SERVER_RESPONSE 2023-08-10 01:09:44,874 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - State RECEIVE_SERVER_RESPONSE at end of evaluating challenge 2023-08-10 01:09:44,876 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Set SASL client state to INTERMEDIATE 2023-08-10 01:09:44,903 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - State RECEIVE_SERVER_RESPONSE at start of evaluating challenge 2023-08-10 01:09:44,905 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Response from server: AuthenticationResponse(version=2020_10_22, requestId=cc725fb1-5e15-4798-8b78-24a39e310274) 2023-08-10 01:09:44,905 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Setting SASL/AWS_MSK_IAM client state to COMPLETE 2023-08-10 01:09:44,966 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - State COMPLETE at end of evaluating challenge 2023-08-10 01:09:44,966 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Set SASL client state to COMPLETE 2023-08-10 01:09:44,967 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Finished authentication with session expiration in 21553098 ms and session re-authentication on or after 20177276 ms 2023-08-10 01:09:44,967 DEBUG o.a.kafka.common.network.Selector akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Successfully authenticated with b-3.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com/10.0.5.125 2023-08-10 01:09:44,967 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Initiating API versions fetch from node 3. 2023-08-10 01:09:44,968 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Sending API_VERSIONS request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=adminclient-2, correlationId=2) and timeout 3600000 to node 3: ApiVersionsRequestData(clientSoftwareName='apache-kafka-java', clientSoftwareVersion='3.2.3') 2023-08-10 01:09:44,977 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Received API_VERSIONS response from node 3 for request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=adminclient-2, correlationId=2): ApiVersionsResponseData(errorCode=0, apiKeys=[ApiVersion(apiKey=0, minVersion=0, maxVersion=9), ApiVersion(apiKey=1, minVersion=0, maxVersion=13), ApiVersion(apiKey=2, minVersion=0, maxVersion=7), ApiVersion(apiKey=3, minVersion=0, maxVersion=12), ApiVersion(apiKey=4, minVersion=0, maxVersion=6), ApiVersion(apiKey=5, minVersion=0, maxVersion=3), ApiVersion(apiKey=6, minVersion=0, maxVersion=7), ApiVersion(apiKey=7, minVersion=0, maxVersion=3), ApiVersion(apiKey=8, minVersion=0, maxVersion=8), ApiVersion(apiKey=9, minVersion=0, maxVersion=8), ApiVersion(apiKey=10, minVersion=0, maxVersion=4), ApiVersion(apiKey=11, minVersion=0, maxVersion=9), ApiVersion(apiKey=12, minVersion=0, maxVersion=4), ApiVersion(apiKey=13, minVersion=0, maxVersion=5), ApiVersion(apiKey=14, minVersion=0, maxVersion=5), ApiVersion(apiKey=15, minVersion=0, maxVersion=5), ApiVersion(apiKey=16, minVersion=0, maxVersion=4), ApiVersion(apiKey=17, minVersion=0, maxVersion=1), ApiVersion(apiKey=18, minVersion=0, maxVersion=3), ApiVersion(apiKey=19, minVersion=0, maxVersion=7), ApiVersion(apiKey=20, minVersion=0, maxVersion=6), ApiVersion(apiKey=21, minVersion=0, maxVersion=2), ApiVersion(apiKey=22, minVersion=0, maxVersion=4), ApiVersion(apiKey=23, minVersion=0, maxVersion=4), ApiVersion(apiKey=24, minVersion=0, maxVersion=3), ApiVersion(apiKey=25, minVersion=0, maxVersion=3), ApiVersion(apiKey=26, minVersion=0, maxVersion=3), ApiVersion(apiKey=27, minVersion=0, maxVersion=1), ApiVersion(apiKey=28, minVersion=0, maxVersion=3), ApiVersion(apiKey=29, minVersion=0, maxVersion=3), ApiVersion(apiKey=30, minVersion=0, maxVersion=3), ApiVersion(apiKey=31, minVersion=0, maxVersion=3), ApiVersion(apiKey=32, minVersion=0, maxVersion=4), ApiVersion(apiKey=33, minVersion=0, maxVersion=2), ApiVersion(apiKey=34, minVersion=0, maxVersion=2), ApiVersion(apiKey=35, minVersion=0, maxVersion=4), ApiVersion(apiKey=36, minVersion=0, maxVersion=2), ApiVersion(apiKey=37, minVersion=0, maxVersion=3), ApiVersion(apiKey=38, minVersion=0, maxVersion=3), ApiVersion(apiKey=39, minVersion=0, maxVersion=2), ApiVersion(apiKey=40, minVersion=0, maxVersion=2), ApiVersion(apiKey=41, minVersion=0, maxVersion=3), ApiVersion(apiKey=42, minVersion=0, maxVersion=2), ApiVersion(apiKey=43, minVersion=0, maxVersion=2), ApiVersion(apiKey=44, minVersion=0, maxVersion=1), ApiVersion(apiKey=45, minVersion=0, maxVersion=0), ApiVersion(apiKey=46, minVersion=0, maxVersion=0), ApiVersion(apiKey=47, minVersion=0, maxVersion=0), ApiVersion(apiKey=48, minVersion=0, maxVersion=1), ApiVersion(apiKey=49, minVersion=0, maxVersion=1), ApiVersion(apiKey=50, minVersion=0, maxVersion=0), ApiVersion(apiKey=51, minVersion=0, maxVersion=0), ApiVersion(apiKey=56, minVersion=0, maxVersion=2), ApiVersion(apiKey=57, minVersion=0, maxVersion=1), ApiVersion(apiKey=60, minVersion=0, maxVersion=0), ApiVersion(apiKey=61, minVersion=0, maxVersion=0), ApiVersion(apiKey=65, minVersion=0, maxVersion=0), ApiVersion(apiKey=66, minVersion=0, maxVersion=0), ApiVersion(apiKey=67, minVersion=0, maxVersion=0)], throttleTimeMs=0, supportedFeatures=[], finalizedFeaturesEpoch=0, finalizedFeatures=[]) 2023-08-10 01:09:44,979 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Node 3 has finalized features epoch: 0, finalized features: [], supported features: [], API versions: (Produce(0): 0 to 9 [usable: 9], Fetch(1): 0 to 13 [usable: 13], ListOffsets(2): 0 to 7 [usable: 7], Metadata(3): 0 to 12 [usable: 12], LeaderAndIsr(4): 0 to 6 [usable: 6], StopReplica(5): 0 to 3 [usable: 3], UpdateMetadata(6): 0 to 7 [usable: 7], ControlledShutdown(7): 0 to 3 [usable: 3], OffsetCommit(8): 0 to 8 [usable: 8], OffsetFetch(9): 0 to 8 [usable: 8], FindCoordinator(10): 0 to 4 [usable: 4], JoinGroup(11): 0 to 9 [usable: 9], Heartbeat(12): 0 to 4 [usable: 4], LeaveGroup(13): 0 to 5 [usable: 5], SyncGroup(14): 0 to 5 [usable: 5], DescribeGroups(15): 0 to 5 [usable: 5], ListGroups(16): 0 to 4 [usable: 4], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 3 [usable: 3], CreateTopics(19): 0 to 7 [usable: 7], DeleteTopics(20): 0 to 6 [usable: 6], DeleteRecords(21): 0 to 2 [usable: 2], InitProducerId(22): 0 to 4 [usable: 4], OffsetForLeaderEpoch(23): 0 to 4 [usable: 4], AddPartitionsToTxn(24): 0 to 3 [usable: 3], AddOffsetsToTxn(25): 0 to 3 [usable: 3], EndTxn(26): 0 to 3 [usable: 3], WriteTxnMarkers(27): 0 to 1 [usable: 1], TxnOffsetCommit(28): 0 to 3 [usable: 3], DescribeAcls(29): 0 to 3 [usable: 2], CreateAcls(30): 0 to 3 [usable: 2], DeleteAcls(31): 0 to 3 [usable: 2], DescribeConfigs(32): 0 to 4 [usable: 4], AlterConfigs(33): 0 to 2 [usable: 2], AlterReplicaLogDirs(34): 0 to 2 [usable: 2], DescribeLogDirs(35): 0 to 4 [usable: 3], SaslAuthenticate(36): 0 to 2 [usable: 2], CreatePartitions(37): 0 to 3 [usable: 3], CreateDelegationToken(38): 0 to 3 [usable: 2], RenewDelegationToken(39): 0 to 2 [usable: 2], ExpireDelegationToken(40): 0 to 2 [usable: 2], DescribeDelegationToken(41): 0 to 3 [usable: 2], DeleteGroups(42): 0 to 2 [usable: 2], ElectLeaders(43): 0 to 2 [usable: 2], IncrementalAlterConfigs(44): 0 to 1 [usable: 1], AlterPartitionReassignments(45): 0 [usable: 0], ListPartitionReassignments(46): 0 [usable: 0], OffsetDelete(47): 0 [usable: 0], DescribeClientQuotas(48): 0 to 1 [usable: 1], AlterClientQuotas(49): 0 to 1 [usable: 1], DescribeUserScramCredentials(50): 0 [usable: 0], AlterUserScramCredentials(51): 0 [usable: 0], AlterPartition(56): 0 to 2 [usable: 1], UpdateFeatures(57): 0 to 1 [usable: 0], DescribeCluster(60): 0 [usable: 0], DescribeProducers(61): 0 [usable: 0], DescribeTransactions(65): 0 [usable: 0], ListTransactions(66): 0 [usable: 0], AllocateProducerIds(67): 0 [usable: 0]). 2023-08-10 01:09:44,980 DEBUG o.a.k.clients.admin.KafkaAdminClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Sending MetadataRequestData(topics=[], allowAutoTopicCreation=true, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false) to b-3.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098 (id: 3 rack: apse2-az2). correlationId=3, timeoutMs=8888 2023-08-10 01:09:45,066 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Sending METADATA request with header RequestHeader(apiKey=METADATA, apiVersion=12, clientId=adminclient-2, correlationId=3) and timeout 8888 to node 3: MetadataRequestData(topics=[], allowAutoTopicCreation=true, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false) 2023-08-10 01:09:45,071 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Received METADATA response from node 3 for request with header RequestHeader(apiKey=METADATA, apiVersion=12, clientId=adminclient-2, correlationId=3): MetadataResponseData(throttleTimeMs=0, brokers=[MetadataResponseBroker(nodeId=2, host='b-2.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com', port=9098, rack='apse2-az3'), MetadataResponseBroker(nodeId=3, host='b-3.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com', port=9098, rack='apse2-az2'), MetadataResponseBroker(nodeId=1, host='b-1.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com', port=9098, rack='apse2-az1')], clusterId='E7DxCekgTJKvUyP3nH8P2w', controllerId=1, topics=[], clusterAuthorizedOperations=-2147483648) 2023-08-10 01:09:45,075 DEBUG o.a.k.clients.admin.KafkaAdminClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Queueing Call(callName=listConsumerGroups, deadlineMs=1691629793868, tries=0, nextAllowedTryMs=0) with a timeout 8794 ms from now. 2023-08-10 01:09:45,076 DEBUG o.a.k.clients.admin.KafkaAdminClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Queueing Call(callName=listConsumerGroups, deadlineMs=1691629793868, tries=0, nextAllowedTryMs=0) with a timeout 8792 ms from now. 2023-08-10 01:09:45,077 DEBUG o.a.k.clients.admin.KafkaAdminClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Queueing Call(callName=listConsumerGroups, deadlineMs=1691629793868, tries=0, nextAllowedTryMs=0) with a timeout 8791 ms from now. 2023-08-10 01:09:45,078 DEBUG o.a.k.clients.admin.KafkaAdminClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Sending ListGroupsRequestData(statesFilter=[]) to b-3.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098 (id: 3 rack: apse2-az2). correlationId=4, timeoutMs=8796 2023-08-10 01:09:45,079 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Sending LIST_GROUPS request with header RequestHeader(apiKey=LIST_GROUPS, apiVersion=4, clientId=adminclient-2, correlationId=4) and timeout 8796 to node 3: ListGroupsRequestData(statesFilter=[]) 2023-08-10 01:09:45,079 DEBUG o.apache.kafka.clients.ClientUtils akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Resolved host b-2.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com as 10.0.3.55 2023-08-10 01:09:45,080 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Initiating connection to node b-2.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098 (id: 2 rack: apse2-az3) using address b-2.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com/10.0.3.55 2023-08-10 01:09:45,080 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Set SASL client state to SEND_APIVERSIONS_REQUEST 2023-08-10 01:09:45,081 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Creating SaslClient: client=null;service=kafka;serviceHostname=b-2.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com;mechs=[AWS_MSK_IAM] 2023-08-10 01:09:45,082 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Setting SASL/AWS_MSK_IAM client state to SEND_CLIENT_FIRST_MESSAGE 2023-08-10 01:09:45,083 DEBUG o.apache.kafka.clients.ClientUtils akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Resolved host b-1.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com as 10.0.1.184 2023-08-10 01:09:45,083 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Initiating connection to node b-1.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098 (id: 1 rack: apse2-az1) using address b-1.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com/10.0.1.184 2023-08-10 01:09:45,084 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Set SASL client state to SEND_APIVERSIONS_REQUEST 2023-08-10 01:09:45,166 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Creating SaslClient: client=null;service=kafka;serviceHostname=b-1.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com;mechs=[AWS_MSK_IAM] 2023-08-10 01:09:45,167 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Setting SASL/AWS_MSK_IAM client state to SEND_CLIENT_FIRST_MESSAGE 2023-08-10 01:09:45,169 DEBUG o.a.kafka.common.network.Selector akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node 2 2023-08-10 01:09:45,273 DEBUG o.a.kafka.common.network.Selector akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node 1 2023-08-10 01:09:45,378 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Completed connection to node 2. Fetching API versions. 2023-08-10 01:09:45,378 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Completed connection to node 1. Fetching API versions. 2023-08-10 01:09:45,568 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Received LIST_GROUPS response from node 3 for request with header RequestHeader(apiKey=LIST_GROUPS, apiVersion=4, clientId=adminclient-2, correlationId=4): ListGroupsResponseData(throttleTimeMs=0, errorCode=0, groups=[]) 2023-08-10 01:09:45,570 DEBUG o.a.k.c.network.SslTransportLayer akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [SslTransportLayer channelId=2 key=channel=java.nio.channels.SocketChannel[connection-pending remote=b-2.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com/10.0.3.55:9098], selector=sun.nio.ch.EPollSelectorImpl@4690ee58, interestOps=8, readyOps=0] SSL handshake completed successfully with peerHost 'b-2.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com' peerPort 9098 peerPrincipal 'CN=*.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com' cipherSuite 'TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384' 2023-08-10 01:09:45,574 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Set SASL client state to RECEIVE_APIVERSIONS_RESPONSE 2023-08-10 01:09:45,574 DEBUG o.a.k.c.network.SslTransportLayer akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [SslTransportLayer channelId=1 key=channel=java.nio.channels.SocketChannel[connection-pending remote=b-1.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com/10.0.1.184:9098], selector=sun.nio.ch.EPollSelectorImpl@4690ee58, interestOps=8, readyOps=0] SSL handshake completed successfully with peerHost 'b-1.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com' peerPort 9098 peerPrincipal 'CN=*.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com' cipherSuite 'TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384' 2023-08-10 01:09:45,575 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Set SASL client state to RECEIVE_APIVERSIONS_RESPONSE 2023-08-10 01:09:45,577 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Set SASL client state to SEND_HANDSHAKE_REQUEST 2023-08-10 01:09:45,578 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Set SASL client state to RECEIVE_HANDSHAKE_RESPONSE 2023-08-10 01:09:45,581 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Set SASL client state to SEND_HANDSHAKE_REQUEST 2023-08-10 01:09:45,582 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Set SASL client state to RECEIVE_HANDSHAKE_RESPONSE 2023-08-10 01:09:45,584 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Set SASL client state to INITIAL 2023-08-10 01:09:45,584 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - State SEND_CLIENT_FIRST_MESSAGE at start of evaluating challenge 2023-08-10 01:09:45,584 DEBUG s.a.m.a.iam.IAMClientCallbackHandler akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Type information for callback: class: software.amazon.msk.auth.iam.internals.AWSCredentialsCallback classloader: jdk.internal.loader.ClassLoaders$AppClassLoader@53ba4f8c from class: software.amazon.msk.auth.iam.IAMClientCallbackHandler classloader: jdk.internal.loader.ClassLoaders$AppClassLoader@53ba4f8c 2023-08-10 01:09:45,584 DEBUG s.a.m.a.iam.IAMClientCallbackHandler akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Selecting provider software.amazon.msk.auth.iam.internals.MSKCredentialProvider to load credentials 2023-08-10 01:09:45,589 DEBUG s.a.m.a.i.i.AWS4SignedPayloadGenerator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Signer configured for kafka-cluster service and ap-southeast-2 region 2023-08-10 01:09:45,666 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Setting SASL/AWS_MSK_IAM client state to RECEIVE_SERVER_RESPONSE 2023-08-10 01:09:45,667 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - State RECEIVE_SERVER_RESPONSE at end of evaluating challenge 2023-08-10 01:09:45,668 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Set SASL client state to INTERMEDIATE 2023-08-10 01:09:45,668 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Set SASL client state to INITIAL 2023-08-10 01:09:45,668 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - State SEND_CLIENT_FIRST_MESSAGE at start of evaluating challenge 2023-08-10 01:09:45,668 DEBUG s.a.m.a.iam.IAMClientCallbackHandler akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Type information for callback: class: software.amazon.msk.auth.iam.internals.AWSCredentialsCallback classloader: jdk.internal.loader.ClassLoaders$AppClassLoader@53ba4f8c from class: software.amazon.msk.auth.iam.IAMClientCallbackHandler classloader: jdk.internal.loader.ClassLoaders$AppClassLoader@53ba4f8c 2023-08-10 01:09:45,668 DEBUG s.a.m.a.iam.IAMClientCallbackHandler akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Selecting provider software.amazon.msk.auth.iam.internals.MSKCredentialProvider to load credentials 2023-08-10 01:09:45,671 DEBUG s.a.m.a.i.i.AWS4SignedPayloadGenerator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Signer configured for kafka-cluster service and ap-southeast-2 region 2023-08-10 01:09:45,673 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Setting SASL/AWS_MSK_IAM client state to RECEIVE_SERVER_RESPONSE 2023-08-10 01:09:45,673 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - State RECEIVE_SERVER_RESPONSE at end of evaluating challenge 2023-08-10 01:09:45,674 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Set SASL client state to INTERMEDIATE 2023-08-10 01:09:45,681 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - State RECEIVE_SERVER_RESPONSE at start of evaluating challenge 2023-08-10 01:09:45,683 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Response from server: AuthenticationResponse(version=2020_10_22, requestId=45b48411-055f-49ef-8785-98167e2e3506) 2023-08-10 01:09:45,683 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Setting SASL/AWS_MSK_IAM client state to COMPLETE 2023-08-10 01:09:45,683 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - State COMPLETE at end of evaluating challenge 2023-08-10 01:09:45,683 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Set SASL client state to COMPLETE 2023-08-10 01:09:45,684 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Finished authentication with session expiration in 21552321 ms and session re-authentication on or after 18971392 ms 2023-08-10 01:09:45,684 DEBUG o.a.kafka.common.network.Selector akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Successfully authenticated with b-2.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com/10.0.3.55 2023-08-10 01:09:45,684 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Initiating API versions fetch from node 2. 2023-08-10 01:09:45,684 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Sending API_VERSIONS request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=adminclient-2, correlationId=5) and timeout 3600000 to node 2: ApiVersionsRequestData(clientSoftwareName='apache-kafka-java', clientSoftwareVersion='3.2.3') 2023-08-10 01:09:45,687 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Received API_VERSIONS response from node 2 for request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=adminclient-2, correlationId=5): ApiVersionsResponseData(errorCode=0, apiKeys=[ApiVersion(apiKey=0, minVersion=0, maxVersion=9), ApiVersion(apiKey=1, minVersion=0, maxVersion=13), ApiVersion(apiKey=2, minVersion=0, maxVersion=7), ApiVersion(apiKey=3, minVersion=0, maxVersion=12), ApiVersion(apiKey=4, minVersion=0, maxVersion=6), ApiVersion(apiKey=5, minVersion=0, maxVersion=3), ApiVersion(apiKey=6, minVersion=0, maxVersion=7), ApiVersion(apiKey=7, minVersion=0, maxVersion=3), ApiVersion(apiKey=8, minVersion=0, maxVersion=8), ApiVersion(apiKey=9, minVersion=0, maxVersion=8), ApiVersion(apiKey=10, minVersion=0, maxVersion=4), ApiVersion(apiKey=11, minVersion=0, maxVersion=9), ApiVersion(apiKey=12, minVersion=0, maxVersion=4), ApiVersion(apiKey=13, minVersion=0, maxVersion=5), ApiVersion(apiKey=14, minVersion=0, maxVersion=5), ApiVersion(apiKey=15, minVersion=0, maxVersion=5), ApiVersion(apiKey=16, minVersion=0, maxVersion=4), ApiVersion(apiKey=17, minVersion=0, maxVersion=1), ApiVersion(apiKey=18, minVersion=0, maxVersion=3), ApiVersion(apiKey=19, minVersion=0, maxVersion=7), ApiVersion(apiKey=20, minVersion=0, maxVersion=6), ApiVersion(apiKey=21, minVersion=0, maxVersion=2), ApiVersion(apiKey=22, minVersion=0, maxVersion=4), ApiVersion(apiKey=23, minVersion=0, maxVersion=4), ApiVersion(apiKey=24, minVersion=0, maxVersion=3), ApiVersion(apiKey=25, minVersion=0, maxVersion=3), ApiVersion(apiKey=26, minVersion=0, maxVersion=3), ApiVersion(apiKey=27, minVersion=0, maxVersion=1), ApiVersion(apiKey=28, minVersion=0, maxVersion=3), ApiVersion(apiKey=29, minVersion=0, maxVersion=3), ApiVersion(apiKey=30, minVersion=0, maxVersion=3), ApiVersion(apiKey=31, minVersion=0, maxVersion=3), ApiVersion(apiKey=32, minVersion=0, maxVersion=4), ApiVersion(apiKey=33, minVersion=0, maxVersion=2), ApiVersion(apiKey=34, minVersion=0, maxVersion=2), ApiVersion(apiKey=35, minVersion=0, maxVersion=4), ApiVersion(apiKey=36, minVersion=0, maxVersion=2), ApiVersion(apiKey=37, minVersion=0, maxVersion=3), ApiVersion(apiKey=38, minVersion=0, maxVersion=3), ApiVersion(apiKey=39, minVersion=0, maxVersion=2), ApiVersion(apiKey=40, minVersion=0, maxVersion=2), ApiVersion(apiKey=41, minVersion=0, maxVersion=3), ApiVersion(apiKey=42, minVersion=0, maxVersion=2), ApiVersion(apiKey=43, minVersion=0, maxVersion=2), ApiVersion(apiKey=44, minVersion=0, maxVersion=1), ApiVersion(apiKey=45, minVersion=0, maxVersion=0), ApiVersion(apiKey=46, minVersion=0, maxVersion=0), ApiVersion(apiKey=47, minVersion=0, maxVersion=0), ApiVersion(apiKey=48, minVersion=0, maxVersion=1), ApiVersion(apiKey=49, minVersion=0, maxVersion=1), ApiVersion(apiKey=50, minVersion=0, maxVersion=0), ApiVersion(apiKey=51, minVersion=0, maxVersion=0), ApiVersion(apiKey=56, minVersion=0, maxVersion=2), ApiVersion(apiKey=57, minVersion=0, maxVersion=1), ApiVersion(apiKey=60, minVersion=0, maxVersion=0), ApiVersion(apiKey=61, minVersion=0, maxVersion=0), ApiVersion(apiKey=65, minVersion=0, maxVersion=0), ApiVersion(apiKey=66, minVersion=0, maxVersion=0), ApiVersion(apiKey=67, minVersion=0, maxVersion=0)], throttleTimeMs=0, supportedFeatures=[], finalizedFeaturesEpoch=0, finalizedFeatures=[]) 2023-08-10 01:09:45,767 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Node 2 has finalized features epoch: 0, finalized features: [], supported features: [], API versions: (Produce(0): 0 to 9 [usable: 9], Fetch(1): 0 to 13 [usable: 13], ListOffsets(2): 0 to 7 [usable: 7], Metadata(3): 0 to 12 [usable: 12], LeaderAndIsr(4): 0 to 6 [usable: 6], StopReplica(5): 0 to 3 [usable: 3], UpdateMetadata(6): 0 to 7 [usable: 7], ControlledShutdown(7): 0 to 3 [usable: 3], OffsetCommit(8): 0 to 8 [usable: 8], OffsetFetch(9): 0 to 8 [usable: 8], FindCoordinator(10): 0 to 4 [usable: 4], JoinGroup(11): 0 to 9 [usable: 9], Heartbeat(12): 0 to 4 [usable: 4], LeaveGroup(13): 0 to 5 [usable: 5], SyncGroup(14): 0 to 5 [usable: 5], DescribeGroups(15): 0 to 5 [usable: 5], ListGroups(16): 0 to 4 [usable: 4], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 3 [usable: 3], CreateTopics(19): 0 to 7 [usable: 7], DeleteTopics(20): 0 to 6 [usable: 6], DeleteRecords(21): 0 to 2 [usable: 2], InitProducerId(22): 0 to 4 [usable: 4], OffsetForLeaderEpoch(23): 0 to 4 [usable: 4], AddPartitionsToTxn(24): 0 to 3 [usable: 3], AddOffsetsToTxn(25): 0 to 3 [usable: 3], EndTxn(26): 0 to 3 [usable: 3], WriteTxnMarkers(27): 0 to 1 [usable: 1], TxnOffsetCommit(28): 0 to 3 [usable: 3], DescribeAcls(29): 0 to 3 [usable: 2], CreateAcls(30): 0 to 3 [usable: 2], DeleteAcls(31): 0 to 3 [usable: 2], DescribeConfigs(32): 0 to 4 [usable: 4], AlterConfigs(33): 0 to 2 [usable: 2], AlterReplicaLogDirs(34): 0 to 2 [usable: 2], DescribeLogDirs(35): 0 to 4 [usable: 3], SaslAuthenticate(36): 0 to 2 [usable: 2], CreatePartitions(37): 0 to 3 [usable: 3], CreateDelegationToken(38): 0 to 3 [usable: 2], RenewDelegationToken(39): 0 to 2 [usable: 2], ExpireDelegationToken(40): 0 to 2 [usable: 2], DescribeDelegationToken(41): 0 to 3 [usable: 2], DeleteGroups(42): 0 to 2 [usable: 2], ElectLeaders(43): 0 to 2 [usable: 2], IncrementalAlterConfigs(44): 0 to 1 [usable: 1], AlterPartitionReassignments(45): 0 [usable: 0], ListPartitionReassignments(46): 0 [usable: 0], OffsetDelete(47): 0 [usable: 0], DescribeClientQuotas(48): 0 to 1 [usable: 1], AlterClientQuotas(49): 0 to 1 [usable: 1], DescribeUserScramCredentials(50): 0 [usable: 0], AlterUserScramCredentials(51): 0 [usable: 0], AlterPartition(56): 0 to 2 [usable: 1], UpdateFeatures(57): 0 to 1 [usable: 0], DescribeCluster(60): 0 [usable: 0], DescribeProducers(61): 0 [usable: 0], DescribeTransactions(65): 0 [usable: 0], ListTransactions(66): 0 [usable: 0], AllocateProducerIds(67): 0 [usable: 0]). 2023-08-10 01:09:45,767 DEBUG o.a.k.clients.admin.KafkaAdminClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Sending ListGroupsRequestData(statesFilter=[]) to b-2.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098 (id: 2 rack: apse2-az3). correlationId=6, timeoutMs=8101 2023-08-10 01:09:45,767 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Sending LIST_GROUPS request with header RequestHeader(apiKey=LIST_GROUPS, apiVersion=4, clientId=adminclient-2, correlationId=6) and timeout 8101 to node 2: ListGroupsRequestData(statesFilter=[]) 2023-08-10 01:09:45,769 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - State RECEIVE_SERVER_RESPONSE at start of evaluating challenge 2023-08-10 01:09:45,770 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Response from server: AuthenticationResponse(version=2020_10_22, requestId=5000f51e-3345-4ed9-9e73-ab93e5c06953) 2023-08-10 01:09:45,770 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Setting SASL/AWS_MSK_IAM client state to COMPLETE 2023-08-10 01:09:45,770 DEBUG s.a.m.a.iam.internals.IAMSaslClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - State COMPLETE at end of evaluating challenge 2023-08-10 01:09:45,771 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Set SASL client state to COMPLETE 2023-08-10 01:09:45,771 DEBUG o.a.k.c.s.a.SaslClientAuthenticator akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Finished authentication with session expiration in 21552309 ms and session re-authentication on or after 18865939 ms 2023-08-10 01:09:45,771 DEBUG o.a.kafka.common.network.Selector akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Successfully authenticated with b-1.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com/10.0.1.184 2023-08-10 01:09:45,771 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Initiating API versions fetch from node 1. 2023-08-10 01:09:45,771 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Sending API_VERSIONS request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=adminclient-2, correlationId=7) and timeout 3600000 to node 1: ApiVersionsRequestData(clientSoftwareName='apache-kafka-java', clientSoftwareVersion='3.2.3') 2023-08-10 01:09:45,773 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Received API_VERSIONS response from node 1 for request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=adminclient-2, correlationId=7): ApiVersionsResponseData(errorCode=0, apiKeys=[ApiVersion(apiKey=0, minVersion=0, maxVersion=9), ApiVersion(apiKey=1, minVersion=0, maxVersion=13), ApiVersion(apiKey=2, minVersion=0, maxVersion=7), ApiVersion(apiKey=3, minVersion=0, maxVersion=12), ApiVersion(apiKey=4, minVersion=0, maxVersion=6), ApiVersion(apiKey=5, minVersion=0, maxVersion=3), ApiVersion(apiKey=6, minVersion=0, maxVersion=7), ApiVersion(apiKey=7, minVersion=0, maxVersion=3), ApiVersion(apiKey=8, minVersion=0, maxVersion=8), ApiVersion(apiKey=9, minVersion=0, maxVersion=8), ApiVersion(apiKey=10, minVersion=0, maxVersion=4), ApiVersion(apiKey=11, minVersion=0, maxVersion=9), ApiVersion(apiKey=12, minVersion=0, maxVersion=4), ApiVersion(apiKey=13, minVersion=0, maxVersion=5), ApiVersion(apiKey=14, minVersion=0, maxVersion=5), ApiVersion(apiKey=15, minVersion=0, maxVersion=5), ApiVersion(apiKey=16, minVersion=0, maxVersion=4), ApiVersion(apiKey=17, minVersion=0, maxVersion=1), ApiVersion(apiKey=18, minVersion=0, maxVersion=3), ApiVersion(apiKey=19, minVersion=0, maxVersion=7), ApiVersion(apiKey=20, minVersion=0, maxVersion=6), ApiVersion(apiKey=21, minVersion=0, maxVersion=2), ApiVersion(apiKey=22, minVersion=0, maxVersion=4), ApiVersion(apiKey=23, minVersion=0, maxVersion=4), ApiVersion(apiKey=24, minVersion=0, maxVersion=3), ApiVersion(apiKey=25, minVersion=0, maxVersion=3), ApiVersion(apiKey=26, minVersion=0, maxVersion=3), ApiVersion(apiKey=27, minVersion=0, maxVersion=1), ApiVersion(apiKey=28, minVersion=0, maxVersion=3), ApiVersion(apiKey=29, minVersion=0, maxVersion=3), ApiVersion(apiKey=30, minVersion=0, maxVersion=3), ApiVersion(apiKey=31, minVersion=0, maxVersion=3), ApiVersion(apiKey=32, minVersion=0, maxVersion=4), ApiVersion(apiKey=33, minVersion=0, maxVersion=2), ApiVersion(apiKey=34, minVersion=0, maxVersion=2), ApiVersion(apiKey=35, minVersion=0, maxVersion=4), ApiVersion(apiKey=36, minVersion=0, maxVersion=2), ApiVersion(apiKey=37, minVersion=0, maxVersion=3), ApiVersion(apiKey=38, minVersion=0, maxVersion=3), ApiVersion(apiKey=39, minVersion=0, maxVersion=2), ApiVersion(apiKey=40, minVersion=0, maxVersion=2), ApiVersion(apiKey=41, minVersion=0, maxVersion=3), ApiVersion(apiKey=42, minVersion=0, maxVersion=2), ApiVersion(apiKey=43, minVersion=0, maxVersion=2), ApiVersion(apiKey=44, minVersion=0, maxVersion=1), ApiVersion(apiKey=45, minVersion=0, maxVersion=0), ApiVersion(apiKey=46, minVersion=0, maxVersion=0), ApiVersion(apiKey=47, minVersion=0, maxVersion=0), ApiVersion(apiKey=48, minVersion=0, maxVersion=1), ApiVersion(apiKey=49, minVersion=0, maxVersion=1), ApiVersion(apiKey=50, minVersion=0, maxVersion=0), ApiVersion(apiKey=51, minVersion=0, maxVersion=0), ApiVersion(apiKey=56, minVersion=0, maxVersion=2), ApiVersion(apiKey=57, minVersion=0, maxVersion=1), ApiVersion(apiKey=60, minVersion=0, maxVersion=0), ApiVersion(apiKey=61, minVersion=0, maxVersion=0), ApiVersion(apiKey=65, minVersion=0, maxVersion=0), ApiVersion(apiKey=66, minVersion=0, maxVersion=0), ApiVersion(apiKey=67, minVersion=0, maxVersion=0)], throttleTimeMs=0, supportedFeatures=[], finalizedFeaturesEpoch=0, finalizedFeatures=[]) 2023-08-10 01:09:45,774 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Node 1 has finalized features epoch: 0, finalized features: [], supported features: [], API versions: (Produce(0): 0 to 9 [usable: 9], Fetch(1): 0 to 13 [usable: 13], ListOffsets(2): 0 to 7 [usable: 7], Metadata(3): 0 to 12 [usable: 12], LeaderAndIsr(4): 0 to 6 [usable: 6], StopReplica(5): 0 to 3 [usable: 3], UpdateMetadata(6): 0 to 7 [usable: 7], ControlledShutdown(7): 0 to 3 [usable: 3], OffsetCommit(8): 0 to 8 [usable: 8], OffsetFetch(9): 0 to 8 [usable: 8], FindCoordinator(10): 0 to 4 [usable: 4], JoinGroup(11): 0 to 9 [usable: 9], Heartbeat(12): 0 to 4 [usable: 4], LeaveGroup(13): 0 to 5 [usable: 5], SyncGroup(14): 0 to 5 [usable: 5], DescribeGroups(15): 0 to 5 [usable: 5], ListGroups(16): 0 to 4 [usable: 4], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 3 [usable: 3], CreateTopics(19): 0 to 7 [usable: 7], DeleteTopics(20): 0 to 6 [usable: 6], DeleteRecords(21): 0 to 2 [usable: 2], InitProducerId(22): 0 to 4 [usable: 4], OffsetForLeaderEpoch(23): 0 to 4 [usable: 4], AddPartitionsToTxn(24): 0 to 3 [usable: 3], AddOffsetsToTxn(25): 0 to 3 [usable: 3], EndTxn(26): 0 to 3 [usable: 3], WriteTxnMarkers(27): 0 to 1 [usable: 1], TxnOffsetCommit(28): 0 to 3 [usable: 3], DescribeAcls(29): 0 to 3 [usable: 2], CreateAcls(30): 0 to 3 [usable: 2], DeleteAcls(31): 0 to 3 [usable: 2], DescribeConfigs(32): 0 to 4 [usable: 4], AlterConfigs(33): 0 to 2 [usable: 2], AlterReplicaLogDirs(34): 0 to 2 [usable: 2], DescribeLogDirs(35): 0 to 4 [usable: 3], SaslAuthenticate(36): 0 to 2 [usable: 2], CreatePartitions(37): 0 to 3 [usable: 3], CreateDelegationToken(38): 0 to 3 [usable: 2], RenewDelegationToken(39): 0 to 2 [usable: 2], ExpireDelegationToken(40): 0 to 2 [usable: 2], DescribeDelegationToken(41): 0 to 3 [usable: 2], DeleteGroups(42): 0 to 2 [usable: 2], ElectLeaders(43): 0 to 2 [usable: 2], IncrementalAlterConfigs(44): 0 to 1 [usable: 1], AlterPartitionReassignments(45): 0 [usable: 0], ListPartitionReassignments(46): 0 [usable: 0], OffsetDelete(47): 0 [usable: 0], DescribeClientQuotas(48): 0 to 1 [usable: 1], AlterClientQuotas(49): 0 to 1 [usable: 1], DescribeUserScramCredentials(50): 0 [usable: 0], AlterUserScramCredentials(51): 0 [usable: 0], AlterPartition(56): 0 to 2 [usable: 1], UpdateFeatures(57): 0 to 1 [usable: 0], DescribeCluster(60): 0 [usable: 0], DescribeProducers(61): 0 [usable: 0], DescribeTransactions(65): 0 [usable: 0], ListTransactions(66): 0 [usable: 0], AllocateProducerIds(67): 0 [usable: 0]). 2023-08-10 01:09:45,774 DEBUG o.a.k.clients.admin.KafkaAdminClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Sending ListGroupsRequestData(statesFilter=[]) to b-1.kafkamessaging.50eowx.c4.kafka.ap-southeast-2.amazonaws.com:9098 (id: 1 rack: apse2-az1). correlationId=8, timeoutMs=8094 2023-08-10 01:09:45,774 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Sending LIST_GROUPS request with header RequestHeader(apiKey=LIST_GROUPS, apiVersion=4, clientId=adminclient-2, correlationId=8) and timeout 8094 to node 1: ListGroupsRequestData(statesFilter=[]) 2023-08-10 01:09:45,867 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Received LIST_GROUPS response from node 2 for request with header RequestHeader(apiKey=LIST_GROUPS, apiVersion=4, clientId=adminclient-2, correlationId=6): ListGroupsResponseData(throttleTimeMs=0, errorCode=0, groups=[]) 2023-08-10 01:09:45,868 DEBUG o.apache.kafka.clients.NetworkClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - [AdminClient clientId=adminclient-2] Received LIST_GROUPS response from node 1 for request with header RequestHeader(apiKey=LIST_GROUPS, apiVersion=4, clientId=adminclient-2, correlationId=8): ListGroupsResponseData(throttleTimeMs=0, errorCode=0, groups=[]) 2023-08-10 01:09:45,972 DEBUG c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-kafka-messaging - Received Offsets Snapshot: Timestamp: 1691629785882 Groups: Earliest Offsets: Topic Partition Earliest Latest Offsets: Topic Partition Offset Last Group Offsets: Group Topic Partition Offset