-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Kafka client Java API wrapper #753
Closed
Closed
Changes from 3 commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
f0f2449
Kafka client Java API wrapper
merlimat e7a49c5
Allow to configure authentication class
merlimat 4feab32
Addressed PR comments
merlimat 0174a7c
Fixed partition naming handling
merlimat 1c42753
Reorganized the modules
merlimat eaca767
Added test to verify the final shaded library
merlimat 5e56490
Fixed parent artifact id
merlimat 61ad39d
Addressed comments
merlimat cf1f71b
Fixed min batch time window exception
merlimat c569419
Clear consumer in case of partial failures
merlimat File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,133 @@ | ||
<?xml version="1.0" encoding="UTF-8"?> | ||
<!-- | ||
|
||
Licensed to the Apache Software Foundation (ASF) under one | ||
or more contributor license agreements. See the NOTICE file | ||
distributed with this work for additional information | ||
regarding copyright ownership. The ASF licenses this file | ||
to you under the Apache License, Version 2.0 (the | ||
"License"); you may not use this file except in compliance | ||
with the License. You may obtain a copy of the License at | ||
|
||
http://www.apache.org/licenses/LICENSE-2.0 | ||
|
||
Unless required by applicable law or agreed to in writing, | ||
software distributed under the License is distributed on an | ||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
KIND, either express or implied. See the License for the | ||
specific language governing permissions and limitations | ||
under the License. | ||
|
||
--> | ||
<project | ||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" | ||
xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> | ||
<modelVersion>4.0.0</modelVersion> | ||
|
||
<parent> | ||
<groupId>org.apache.pulsar</groupId> | ||
<artifactId>pulsar</artifactId> | ||
<version>1.20.0-incubating-SNAPSHOT</version> | ||
<relativePath>..</relativePath> | ||
</parent> | ||
|
||
<artifactId>pulsar-client-kafka-compat</artifactId> | ||
<name>Pulsar Kafka compatibility API</name> | ||
|
||
<description>Drop-in replacement for Kafka client library that publishes and consumes | ||
messages on Pulsar topics</description> | ||
|
||
<properties> | ||
<kafka.version>0.10.2.1</kafka.version> | ||
</properties> | ||
|
||
<dependencies> | ||
<dependency> | ||
<groupId>${project.groupId}</groupId> | ||
<artifactId>pulsar-broker</artifactId> | ||
<version>${project.version}</version> | ||
<scope>test</scope> | ||
</dependency> | ||
|
||
<dependency> | ||
<groupId>${project.groupId}</groupId> | ||
<artifactId>pulsar-broker</artifactId> | ||
<version>${project.version}</version> | ||
<scope>test</scope> | ||
<type>test-jar</type> | ||
</dependency> | ||
|
||
<dependency> | ||
<groupId>${project.groupId}</groupId> | ||
<artifactId>managed-ledger</artifactId> | ||
<version>${project.version}</version> | ||
<scope>test</scope> | ||
<type>test-jar</type> | ||
</dependency> | ||
|
||
<dependency> | ||
<groupId>${project.groupId}</groupId> | ||
<artifactId>pulsar-client</artifactId> | ||
<version>${project.version}</version> | ||
</dependency> | ||
|
||
<dependency> | ||
<groupId>org.apache.kafka</groupId> | ||
<artifactId>kafka-clients</artifactId> | ||
<version>${kafka.version}</version> | ||
</dependency> | ||
</dependencies> | ||
|
||
<build> | ||
<plugins> | ||
<plugin> | ||
<groupId>org.apache.maven.plugins</groupId> | ||
<artifactId>maven-shade-plugin</artifactId> | ||
<executions> | ||
<execution> | ||
<phase>package</phase> | ||
<goals> | ||
<goal>shade</goal> | ||
</goals> | ||
<configuration> | ||
<createDependencyReducedPom>true</createDependencyReducedPom> | ||
<promoteTransitiveDependencies>true</promoteTransitiveDependencies> | ||
<artifactSet> | ||
<includes> | ||
<include>org.apache.kafka:kafka-clients</include> | ||
</includes> | ||
</artifactSet> | ||
<filters> | ||
<filter> | ||
<artifact>org.apache.kafka:kafka-clients</artifact> | ||
<excludes> | ||
<exclude>org/apache/kafka/clients/producer/internals/**</exclude> | ||
<exclude>org/apache/kafka/clients/consumer/internals/**</exclude> | ||
</excludes> | ||
</filter> | ||
</filters> | ||
<relocations> | ||
<relocation> | ||
<pattern>org.apache.kafka.clients.producer.KafkaProducer</pattern> | ||
<shadedPattern>org.apache.kafka.clients.producer.OriginalKafkaProducer</shadedPattern> | ||
</relocation> | ||
<relocation> | ||
<pattern>org.apache.kafka.clients.producer.PulsarKafkaProducer</pattern> | ||
<shadedPattern>org.apache.kafka.clients.producer.KafkaProducer</shadedPattern> | ||
</relocation> | ||
<relocation> | ||
<pattern>org.apache.kafka.clients.consumer.KafkaConsumer</pattern> | ||
<shadedPattern>org.apache.kafka.clients.consumer.OriginalKafkaConsumer</shadedPattern> | ||
</relocation> | ||
<relocation> | ||
<pattern>org.apache.kafka.clients.consumer.PulsarKafkaConsumer</pattern> | ||
<shadedPattern>org.apache.kafka.clients.consumer.KafkaConsumer</shadedPattern> | ||
</relocation> | ||
</relocations> | ||
</configuration> | ||
</execution> | ||
</executions> | ||
</plugin> | ||
</plugins> | ||
</build> | ||
</project> |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
just out of curiosity - that means the PulsarKafkaProducer needs to have exactly same constructors as the original KafkaProducer, and same applied for consumer, right? How hard is to keep such compatibility going forward?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, we need to use the same constructor as the original class.
I think we can maintain compatibility up to a specific
kafka-clients
version. That should be noted in the documentation. So far, I haven't seen differences in the constructors from 0.9 to 0.11 versions.In case Kafka adds a new constructor in next releases, the client app would still work unless it tries to use the constructor form.