Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

KafkaIO SDF: Fetch end position for each topic-partition tuple in a background thread, reusing kafka consumers. #32558

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

KafkaIO SDF: Fetch end position for each topic-partition tuple in a b…

6d9857b
Select commit
Loading
Failed to load commit list.
Sign in for the full log view
Open

KafkaIO SDF: Fetch end position for each topic-partition tuple in a background thread, reusing kafka consumers. #32558

KafkaIO SDF: Fetch end position for each topic-partition tuple in a b…
6d9857b
Select commit
Loading
Failed to load commit list.
GitHub Actions / Test Results succeeded Oct 2, 2024 in 0s

All 148 tests pass, 1 skipped in 59m 3s

 36 files  + 36   36 suites  +36   59m 3s ⏱️ + 59m 3s
149 tests +149  148 ✅ +148   1 💤 + 1  0 ❌ ±0 
660 runs  +660  650 ✅ +650  10 💤 +10  0 ❌ ±0 

Results for commit 6d9857b. ± Comparison against earlier commit 9f8476da.

Annotations

Check notice on line 0 in .github

See this annotation in the file changed.

@github-actions github-actions / Test Results

1 skipped test found 

There is 1 skipped test, see "Raw output" for the name of the skipped test.
Raw output 
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testUnboundedSourceWithoutBoundedWrapper

Check notice on line 0 in .github

See this annotation in the file changed.

@github-actions github-actions / Test Results

149 tests found 

There are 149 tests, see "Raw output" for the full list of tests.
Raw output 
org.apache.beam.sdk.io.kafka.ConfluentSchemaRegistryDeserializerProviderTest ‑ testDeserialize
org.apache.beam.sdk.io.kafka.ConfluentSchemaRegistryDeserializerProviderTest ‑ testGetCoder
org.apache.beam.sdk.io.kafka.CustomTimestampPolicyWithLimitedDelayTest ‑ testCustomTimestampPolicyWithLimitedDelay
org.apache.beam.sdk.io.kafka.KafkaCommitOffsetTest ‑ testCommitOffsetDoFn
org.apache.beam.sdk.io.kafka.KafkaCommitOffsetTest ‑ testCommitOffsetError
org.apache.beam.sdk.io.kafka.KafkaCommitOffsetTest ‑ testKafkaOffsetComposite
org.apache.beam.sdk.io.kafka.KafkaCommitOffsetTest ‑ testKafkaOffsetCompositeLegacy
org.apache.beam.sdk.io.kafka.KafkaDlqTest ‑ testKafkaErrorFnFailure
org.apache.beam.sdk.io.kafka.KafkaDlqTest ‑ testKafkaErrorFnSuccess
org.apache.beam.sdk.io.kafka.KafkaIOExternalTest ‑ testConstructKafkaRead
org.apache.beam.sdk.io.kafka.KafkaIOExternalTest ‑ testConstructKafkaReadWithoutMetadata
org.apache.beam.sdk.io.kafka.KafkaIOExternalTest ‑ testConstructKafkaWrite
org.apache.beam.sdk.io.kafka.KafkaIOExternalTest ‑ testKafkaRecordToExternalKafkaRecord
org.apache.beam.sdk.io.kafka.KafkaIOExternalTest ‑ testKafkaRecordToExternalKafkaRecordWithNullKeyAndValue
org.apache.beam.sdk.io.kafka.KafkaIOIT ‑ testKafkaIOExternalRoundtripWithMetadataAndNullKeysAndValues
org.apache.beam.sdk.io.kafka.KafkaIOIT ‑ testKafkaIOReadsAndWritesCorrectlyInBatch
org.apache.beam.sdk.io.kafka.KafkaIOIT ‑ testKafkaIOSDFReadWithErrorHandler
org.apache.beam.sdk.io.kafka.KafkaIOIT ‑ testKafkaIOSDFResumesCorrectly
org.apache.beam.sdk.io.kafka.KafkaIOIT ‑ testKafkaIOWriteWithErrorHandler
org.apache.beam.sdk.io.kafka.KafkaIOIT ‑ testKafkaViaManagedSchemaTransformAvro
org.apache.beam.sdk.io.kafka.KafkaIOIT ‑ testKafkaViaManagedSchemaTransformJson
org.apache.beam.sdk.io.kafka.KafkaIOIT ‑ testKafkaWithDelayedStopReadingFunction
org.apache.beam.sdk.io.kafka.KafkaIOIT ‑ testKafkaWithDynamicPartitions
org.apache.beam.sdk.io.kafka.KafkaIOIT ‑ testKafkaWithStopReadingFunction
org.apache.beam.sdk.io.kafka.KafkaIOIT ‑ testWatermarkUpdateWithSparseMessages
org.apache.beam.sdk.io.kafka.KafkaIOReadImplementationCompatibilityTest ‑ testKafkaIOReadPropertiesEnumValuePresence
org.apache.beam.sdk.io.kafka.KafkaIOReadImplementationCompatibilityTest ‑ testPrimitiveKafkaIOReadPropertiesDefaultValueExistence
org.apache.beam.sdk.io.kafka.KafkaIOReadImplementationCompatibilityTest ‑ testReadTransformCreationWithBothImplementationBoundProperties
org.apache.beam.sdk.io.kafka.KafkaIOReadImplementationCompatibilityTest ‑ testReadTransformCreationWithLegacyImplementationBoundProperty
org.apache.beam.sdk.io.kafka.KafkaIOReadImplementationCompatibilityTest ‑ testReadTransformCreationWithSdfImplementationBoundProperty
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ noWarningsWithNoAllowDuplicatesAndCommitOffsets
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testDeserializationWithHeaders
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testDisableRedistributeKafkaOffsetLegacy
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testExactlyOnceSink
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testExactlyOnceSinkWithSendException
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testKafkaWriteHeaders
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testNumKeysIgnoredWithRedistributeNotEnabled
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testReadAvroGenericRecordsWithConfluentSchemaRegistry
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testReadAvroSpecificRecordsWithConfluentSchemaRegistry
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testRecordsSink
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testResolveDefaultApiTimeout
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testRiskyConfigurationWarnsProperly
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testSink
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testSinkDisplayData
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testSinkMetrics
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testSinkProducerRecordsWithCustomPartition
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testSinkProducerRecordsWithCustomTS
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testSinkToMultipleTopics
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testSinkWithSendErrors
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testSinkWithSerializationErrors
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testSourceDisplayData
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testSourceWithExplicitPartitionsDisplayData
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testSourceWithPatternDisplayData
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testUnboundedReaderLogsCommitFailure
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testUnboundedSource
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testUnboundedSourceCheckpointMark
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testUnboundedSourceCheckpointMarkWithEmptyPartitions
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testUnboundedSourceCreateTimestamps
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testUnboundedSourceCustomTimestamps
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testUnboundedSourceLogAppendTimestamps
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testUnboundedSourceMetrics
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testUnboundedSourceRawSizeMetric
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testUnboundedSourceSplits
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testUnboundedSourceStartReadTime
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testUnboundedSourceStartReadTimeException
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testUnboundedSourceTimestamps
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testUnboundedSourceWithExceptionInKafkaFetch
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testUnboundedSourceWithExplicitPartitions
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testUnboundedSourceWithPartiallyMatchedPattern
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testUnboundedSourceWithPattern
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testUnboundedSourceWithSingleTopic
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testUnboundedSourceWithUnmatchedPattern
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testUnboundedSourceWithWrongTopic
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testUnboundedSourceWithoutBoundedWrapper
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testUnreachableKafkaBrokers
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testValuesSink
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testWithInvalidConsumerPollingTimeout
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ testWithValidConsumerPollingTimeout
org.apache.beam.sdk.io.kafka.KafkaIOTest ‑ warningsWithAllowDuplicatesEnabledAndCommitOffsets
org.apache.beam.sdk.io.kafka.KafkaIOUtilsTest ‑ testOffsetConsumerConfigOverrides
org.apache.beam.sdk.io.kafka.KafkaReadSchemaTransformProviderTest ‑ testBuildTransformWithAvroSchema
org.apache.beam.sdk.io.kafka.KafkaReadSchemaTransformProviderTest ‑ testBuildTransformWithJsonSchema
org.apache.beam.sdk.io.kafka.KafkaReadSchemaTransformProviderTest ‑ testBuildTransformWithManaged
org.apache.beam.sdk.io.kafka.KafkaReadSchemaTransformProviderTest ‑ testBuildTransformWithProtoFormat
org.apache.beam.sdk.io.kafka.KafkaReadSchemaTransformProviderTest ‑ testBuildTransformWithProtoFormatWrongMessageName
org.apache.beam.sdk.io.kafka.KafkaReadSchemaTransformProviderTest ‑ testBuildTransformWithProtoSchemaFormat
org.apache.beam.sdk.io.kafka.KafkaReadSchemaTransformProviderTest ‑ testBuildTransformWithRawFormat
org.apache.beam.sdk.io.kafka.KafkaReadSchemaTransformProviderTest ‑ testBuildTransformWithoutProtoSchemaFormat
org.apache.beam.sdk.io.kafka.KafkaReadSchemaTransformProviderTest ‑ testFindTransformAndMakeItWork
org.apache.beam.sdk.io.kafka.KafkaReadSchemaTransformProviderTest ‑ testManagedMappings
org.apache.beam.sdk.io.kafka.KafkaReadSchemaTransformProviderTest ‑ testValidConfigurations
org.apache.beam.sdk.io.kafka.KafkaRecordCoderTest ‑ testCoderIsSerializableWithWellKnownCoderType
org.apache.beam.sdk.io.kafka.KafkaRecordCoderTest ‑ testKafkaRecordSerializableWithHeaders
org.apache.beam.sdk.io.kafka.KafkaRecordCoderTest ‑ testKafkaRecordSerializableWithNullValueHeader
org.apache.beam.sdk.io.kafka.KafkaRecordCoderTest ‑ testKafkaRecordSerializableWithoutHeaders
org.apache.beam.sdk.io.kafka.KafkaSchemaTransformTranslationTest ‑ testReCreateReadTransformFromRow
org.apache.beam.sdk.io.kafka.KafkaSchemaTransformTranslationTest ‑ testReadTransformProtoTranslation
org.apache.beam.sdk.io.kafka.KafkaSchemaTransformTranslationTest ‑ testRecreateWriteTransformFromRow
org.apache.beam.sdk.io.kafka.KafkaSchemaTransformTranslationTest ‑ testWriteTransformProtoTranslation
org.apache.beam.sdk.io.kafka.KafkaWriteSchemaTransformProviderTest ‑ testBuildTransformWithManaged
org.apache.beam.sdk.io.kafka.KafkaWriteSchemaTransformProviderTest ‑ testKafkaErrorFnProtoSuccess
org.apache.beam.sdk.io.kafka.KafkaWriteSchemaTransformProviderTest ‑ testKafkaErrorFnRawSuccess
org.apache.beam.sdk.io.kafka.KafkaWriteSchemaTransformProviderTest ‑ testKafkaErrorFnSuccess
org.apache.beam.sdk.io.kafka.KafkaWriteSchemaTransformProviderTest ‑ testManagedMappings
org.apache.beam.sdk.io.kafka.LocalDeserializerProviderTest ‑ testInferKeyCoder
org.apache.beam.sdk.io.kafka.LocalDeserializerProviderTest ‑ testInferKeyCoderFailure
org.apache.beam.sdk.io.kafka.ProducerRecordCoderTest ‑ testCoderIsSerializableWithWellKnownCoderType
org.apache.beam.sdk.io.kafka.ProducerRecordCoderTest ‑ testProducerRecordSerializableWithHeaders
org.apache.beam.sdk.io.kafka.ProducerRecordCoderTest ‑ testProducerRecordSerializableWithPartition
org.apache.beam.sdk.io.kafka.ProducerRecordCoderTest ‑ testProducerRecordSerializableWithTimestamp
org.apache.beam.sdk.io.kafka.ProducerRecordCoderTest ‑ testProducerRecordSerializableWithoutHeaders
org.apache.beam.sdk.io.kafka.ProducerRecordCoderTest ‑ testProducerRecordSerializableWithoutPartition
org.apache.beam.sdk.io.kafka.ProducerRecordCoderTest ‑ testProducerRecordSerializableWithoutTimestamp
org.apache.beam.sdk.io.kafka.ProducerRecordCoderTest ‑ testProducerRecordStructuralValueWithHeadersApi
org.apache.beam.sdk.io.kafka.ProducerRecordCoderTest ‑ testProducerRecordStructuralValueWithoutHeadersApi
org.apache.beam.sdk.io.kafka.ReadFromKafkaDoFnTest ‑ testBounded
org.apache.beam.sdk.io.kafka.ReadFromKafkaDoFnTest ‑ testConstructorWithPollTimeout
org.apache.beam.sdk.io.kafka.ReadFromKafkaDoFnTest ‑ testInitialRestrictionWhenHasStartOffset
org.apache.beam.sdk.io.kafka.ReadFromKafkaDoFnTest ‑ testInitialRestrictionWhenHasStartTime
org.apache.beam.sdk.io.kafka.ReadFromKafkaDoFnTest ‑ testInitialRestrictionWhenHasStopOffset
org.apache.beam.sdk.io.kafka.ReadFromKafkaDoFnTest ‑ testInitialRestrictionWhenHasStopTime
org.apache.beam.sdk.io.kafka.ReadFromKafkaDoFnTest ‑ testInitialRestrictionWithConsumerPosition
org.apache.beam.sdk.io.kafka.ReadFromKafkaDoFnTest ‑ testInitialRestrictionWithException
org.apache.beam.sdk.io.kafka.ReadFromKafkaDoFnTest ‑ testProcessElement
org.apache.beam.sdk.io.kafka.ReadFromKafkaDoFnTest ‑ testProcessElementWhenTopicPartitionIsRemoved
org.apache.beam.sdk.io.kafka.ReadFromKafkaDoFnTest ‑ testProcessElementWhenTopicPartitionIsStopped
org.apache.beam.sdk.io.kafka.ReadFromKafkaDoFnTest ‑ testProcessElementWithDeserializationExceptionDefaultRecordHandler
org.apache.beam.sdk.io.kafka.ReadFromKafkaDoFnTest ‑ testProcessElementWithDeserializationExceptionRecordingRecordHandler
org.apache.beam.sdk.io.kafka.ReadFromKafkaDoFnTest ‑ testProcessElementWithEarlierOffset
org.apache.beam.sdk.io.kafka.ReadFromKafkaDoFnTest ‑ testProcessElementWithEmptyPoll
org.apache.beam.sdk.io.kafka.ReadFromKafkaDoFnTest ‑ testProcessElementWithException
org.apache.beam.sdk.io.kafka.ReadFromKafkaDoFnTest ‑ testRawSizeMetric
org.apache.beam.sdk.io.kafka.ReadFromKafkaDoFnTest ‑ testSDFCommitOffsetEnabled
org.apache.beam.sdk.io.kafka.ReadFromKafkaDoFnTest ‑ testSDFCommitOffsetNotEnabled
org.apache.beam.sdk.io.kafka.ReadFromKafkaDoFnTest ‑ testUnbounded
org.apache.beam.sdk.io.kafka.TopicPartitionCoderTest ‑ testEncodeDecodeRoundTrip
org.apache.beam.sdk.io.kafka.TopicPartitionCoderTest ‑ testToString
org.apache.beam.sdk.io.kafka.WatchForKafkaTopicPartitionsTest ‑ testGetAllTopicPartitions
org.apache.beam.sdk.io.kafka.WatchForKafkaTopicPartitionsTest ‑ testGetAllTopicPartitionsWithGivenPattern
org.apache.beam.sdk.io.kafka.WatchForKafkaTopicPartitionsTest ‑ testGetAllTopicPartitionsWithGivenTopics
org.apache.beam.sdk.io.kafka.WatchForKafkaTopicPartitionsTest ‑ testPartitionGrowth
org.apache.beam.sdk.io.kafka.WatchForKafkaTopicPartitionsTest ‑ testPartitionSingle
org.apache.beam.sdk.io.kafka.upgrade.KafkaIOTranslationTest ‑ testReCreateReadTransformFromRow
org.apache.beam.sdk.io.kafka.upgrade.KafkaIOTranslationTest ‑ testReCreateReadTransformWithTopics
org.apache.beam.sdk.io.kafka.upgrade.KafkaIOTranslationTest ‑ testReCreateWriteTransformFromRow
org.apache.beam.sdk.io.kafka.upgrade.KafkaIOTranslationTest ‑ testReadTransformRowIncludesAllFields
org.apache.beam.sdk.io.kafka.upgrade.KafkaIOTranslationTest ‑ testReadTransformURNDiscovery
org.apache.beam.sdk.io.kafka.upgrade.KafkaIOTranslationTest ‑ testWriteTransformRowIncludesAllFields
org.apache.beam.sdk.io.kafka.upgrade.KafkaIOTranslationTest ‑ testWriteTransformURNDiscovery
View more details on GitHub Actions