docs: add pubsub develop example for kafka #7059

bzp2010 · 2022-05-17T03:44:35Z

Description

Add pubsub develop documentation example for Apache Kafka.

Checklist

I have explained the need for this PR and the problem it solves
I have explained the changes or the new features added to this PR
I have added tests corresponding to this change
I have updated the documentation to reflect this change
I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)

spacewander · 2022-05-17T07:45:50Z

docs/en/latest/pubsub.md

+message PubSubReq {
+    int64 sequence = 1;
+    oneof req {
+        CmdKafkaFetch cmd_kafka_fetch = 31;


Could you update the code?

updated to source file link

guitu168 · 2022-05-17T11:57:28Z

docs/en/latest/pubsub.md

+:::note Callback function prototype
+The `params` is the data in the protocol definition; the first return value is the data, which needs to contain the fields in the response body definition, and returns the `nil` value when there is an error; the second return value is the error, and returns the error string when there is an error
+:::


Suggested change

:::note Callback function prototype

The `params` is the data in the protocol definition; the first return value is the data, which needs to contain the fields in the response body definition, and returns the `nil` value when there is an error; the second return value is the error, and returns the error string when there is an error

:::

:::note Callback function prototype

The `params` is the data in the protocol definition; the first return value is the data, which needs to contain the fields in the response body definition, and returns the `nil` value when there is an error; the second return value is the error, and returns the error string when there is an error

:::

guitu168 · 2022-05-17T11:58:19Z

docs/en/latest/pubsub.md

+
+```shell
+curl -X PUT 'http://127.0.0.1:9080/apisix/admin/routes/kafka' \
+    -H 'X-API-KEY: <api-key>' \


Suggested change

-H 'X-API-KEY: <api-key>' \

-H 'X-API-KEY: ${api-key}' \

tzssangglass · 2022-05-17T13:07:35Z

docs/en/latest/pubsub.md

+The `params` is the data in the protocol definition; the first return value is the data, which needs to contain the fields in the response body definition, and returns the `nil` value when there is an error; the second return value is the error, and returns the error string when there is an error
+:::
+
+Finally, it enters the loop to wait for client commands and when an error occurs it returns the error and stops the processing flow.


Suggested change

Finally, it enters the loop to wait for client commands and when an error occurs it returns the error and stops the processing flow.

Finally, it enters the loop to wait for client commands, and when an error occurs, it returns the error and stops the processing flow.

tzssangglass · 2022-05-17T13:11:28Z

docs/en/latest/pubsub.md

+
+#### Add a new `scheme` judgment branch to `http_access_phase`
+
+Add a `scheme` judgment branch to the `http_access_phase` function in `apisix/init.lua` to support the processing of `kafka` type upstreams. Because of Apache Kafka has its own clustering and partition scheme, we do not need to use the Apache APISIX built-in load balancing algorithm, so we intercept and take over the processing flow before selecting the upstream node, here using the `kafka_access_phase` function.


Suggested change

Add a `scheme` judgment branch to the `http_access_phase` function in `apisix/init.lua` to support the processing of `kafka` type upstreams. Because of Apache Kafka has its own clustering and partition scheme, we do not need to use the Apache APISIX built-in load balancing algorithm, so we intercept and take over the processing flow before selecting the upstream node, here using the `kafka_access_phase` function.

Add a `scheme` judgment branch to the `http_access_phase` function in `apisix/init.lua` to support the processing of `kafka` type upstreams. Because Apache Kafka has its clustering and partition scheme, we do not need to use the Apache APISIX built-in load balancing algorithm, so we intercept and take over the processing flow before selecting the upstream node, using the `kafka_access_phase` function.

tzssangglass · 2022-05-17T13:15:02Z

docs/en/latest/pubsub.md

+
+First, create an instance of the `pubsub` module, which is provided in the `core` package.
+
+Then, an instance of the Apache Kafka client is created, and this code is omitted here.


Suggested change

Then, an instance of the Apache Kafka client is created, and this code is omitted here.

Then, an instance of the Apache Kafka client is created and omitted code here.

* upsgrteam/master: (351 commits) fix(proxy-cache): bypass when method mismatch cache_method (apache#7111) chore(script): support to install dependencies under arm64 (apache#7091) chore(ci): use the latest build script for apisix-base (apache#7090) fix(batch-requests): ignore "unix:" in the configuration (apache#7106) fix: install dependencies issues (apache#7092) feat(ops): use lua libs to backup config file insteadof shell command (apache#7048) test: reduce CI failure caused by flaky tests (apache#7085) chore(ci): move set_dns.sh to ci dir (apache#7089) feat: release 2.14.0 (apache#7057) docs: update "Tracers" Plugins (apache#7086) docs: update "Traffic" Plugin docs 3 (apache#7064) docs: update "Serverless" Plugins (apache#7076) feat(ops): check process running with posix.signal insteadof lsof (apache#7006) docs: modify how-to-build filename (apache#7087) docs: fix link of hot-reload in docs (apache#7081) chore(ci): apt update before install (apache#7080) docs: add pubsub develop example for kafka (apache#7059) ci: enable rebase in some situation (apache#7074) fix: redirect http to https but port not change (apache#7065) ci: make it pass under OpenResty 1.21 (apache#7067) ...

docs: add develop documentation example for kafka

5d27e9d

bzp2010 requested review from spacewander, membphis, shuaijinchao and tzssangglass May 17, 2022 03:44

bzp2010 self-assigned this May 17, 2022

spacewander reviewed May 17, 2022

View reviewed changes

bzp2010 added 2 commits May 17, 2022 16:06

docs: update source to file link

2e7e755

fix: lint

9e378ee

bzp2010 requested a review from spacewander May 17, 2022 08:57

guitu168 reviewed May 17, 2022

View reviewed changes

tzssangglass reviewed May 17, 2022

View reviewed changes

fix: lint

63794f2

bzp2010 requested a review from tzssangglass May 18, 2022 04:01

spacewander requested a review from juzhiyuan May 18, 2022 07:33

tzssangglass approved these changes May 18, 2022

View reviewed changes

juzhiyuan approved these changes May 18, 2022

View reviewed changes

spacewander merged commit 260529a into apache:master May 19, 2022

Liu-Junlin pushed a commit to Liu-Junlin/apisix that referenced this pull request May 20, 2022

docs: add pubsub develop example for kafka (apache#7059)

af8c16c

spacewander pushed a commit that referenced this pull request Jun 30, 2022

docs: add pubsub develop example for kafka (#7059)

247cd46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: add pubsub develop example for kafka #7059

docs: add pubsub develop example for kafka #7059

bzp2010 commented May 17, 2022

spacewander May 17, 2022

bzp2010 May 17, 2022

guitu168 May 17, 2022

bzp2010 May 18, 2022

guitu168 May 17, 2022

bzp2010 May 18, 2022

tzssangglass May 17, 2022

bzp2010 May 18, 2022

tzssangglass May 17, 2022

bzp2010 May 18, 2022

tzssangglass May 17, 2022

bzp2010 May 18, 2022

	Finally, it enters the loop to wait for client commands and when an error occurs it returns the error and stops the processing flow.
	Finally, it enters the loop to wait for client commands, and when an error occurs, it returns the error and stops the processing flow.


		#### Add a new `scheme` judgment branch to `http_access_phase`

		Add a `scheme` judgment branch to the `http_access_phase` function in `apisix/init.lua` to support the processing of `kafka` type upstreams. Because of Apache Kafka has its own clustering and partition scheme, we do not need to use the Apache APISIX built-in load balancing algorithm, so we intercept and take over the processing flow before selecting the upstream node, here using the `kafka_access_phase` function.


		First, create an instance of the `pubsub` module, which is provided in the `core` package.

		Then, an instance of the Apache Kafka client is created, and this code is omitted here.

docs: add pubsub develop example for kafka #7059

docs: add pubsub develop example for kafka #7059

Conversation

bzp2010 commented May 17, 2022

Description

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment