diff --git a/connectors/aws-eventbridge-sink/v2.10.4.3/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v2.10.4.3/aws-eventbridge-sink.md deleted file mode 100644 index 1550f0ab8..000000000 --- a/connectors/aws-eventbridge-sink/v2.10.4.3/aws-eventbridge-sink.md +++ /dev/null @@ -1,499 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: /images/connectors/aws-eventbridge.png -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: true -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v2.10.4.3/docs/aws-eventbridge-sink.png) - -# Features - -This section describes the features of the AWS EventBridge sink connector. For details about how to configure these -features, see [how to configure](#how-to-configure). - -## Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -{% callout title="Note" type="note" %} -Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the -support of the Sink downstream system. -{% /callout %} - -## Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -## Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -## Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. -> - -## Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -## Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -# How to get - -This section describes how to get the AWS EventBridge sink connector. - -## Work with Function Worker - -You can get the AWS EventBridge sink connector using one of the following methods if you -use [Pulsar Function Worker](https://pulsar.apache.org/docs/en/functions-worker/) to run the connector in a cluster. - -- Download the NAR package - from [the download page](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases/). - -- Build it from the source code. - -To build the AWS EventBridge sink connector from the source code, follow these steps. - -1. Clone the source code to your machine. - - ```bash - git clone https://github.com/streamnative/pulsar-io-aws-eventbridge - ``` - -2. Build the connector in the `pulsar-io-aws-eventbridge` directory. - - ```bash - mvn clean install -DskipTests - ``` - - After the connector is successfully built, a `NAR` package is generated under the target directory. - - ```bash - ls target - pulsar-io-aws-eventbridge-2.10.4.3.nar - ``` - -## Work with Function Mesh - -You can pull the AWS EventBridge sink connector Docker image from -the [Docker Hub](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge) if you -use [Function Mesh](https://functionmesh.io/docs/connectors/run-connector) to run the connector. - -# How to configure - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | No | "" (empty string) | The EventBridge access key ID. | -| `secretAccessKey` | String | no | "" (empty string) | The EventBridge secret access key. | -| `role` | String | no | "" (empty string) | The AWS role to use. | -| `roleSessionName` | String | no | "" (empty string) | The AWS role session name to use. | -| `stsEndpoint` | String | no | "" (empty string) | The STS endpoint to use. By default, the [default STS endpoint](https://sts.amazonaws.com) is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | no | "" (empty string) | The STS region to use, By default, the 'region' config or env region is used. | -| `region` | String | yes | "" (empty string) | The EventBridge region. | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -## Work with Function Worker - -You can create a configuration file (JSON or YAML) to set the properties if you -use [Pulsar Function Worker](https://pulsar.apache.org/docs/en/functions-worker/) to run connectors in a cluster. - -**Example** - -* JSON - - ```json - { - "name": "eventbridge-sink", - "archive": "connectors/pulsar-io-aws-eventbridge-2.10.4.3.nar", - "tenant": "public", - "namespace": "default", - "inputs": [ - "test-aws-eventbridge-pulsar" - ], - "parallelism": 1, - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } - } - ``` - -* YAML - - ```yaml - name: eventbridge-sink - archive: 'connectors/pulsar-io-aws-eventbridge-2.10.4.3.nar' - tenant: public - namespace: default - inputs: - - test-aws-eventbridge-pulsar - parallelism: 1 - configs: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - ``` - -## Work with Function Mesh - -You can create -a [CustomResourceDefinitions (CRD)](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) -to create an AWS EventBridge sink connector. Using CRD makes Function Mesh naturally integrate with the Kubernetes -ecosystem. For more information about Pulsar sink CRD configurations, -see [sink CRD configurations](https://functionmesh.io/docs/connectors/io-crd-config/sink-crd-config). - -You can define a CRD file (YAML) to set the properties as below. - -```yaml -apiVersion: compute.functionmesh.io/v1alpha1 -kind: Sink -metadata: - name: aws-eventbridge-sink-sample -spec: - image: streamnative/pulsar-io-aws-eventbridge:2.10.4.3 - replicas: 1 - maxReplicas: 1 - input: - topics: - - persistent://public/default/test-aws-eventbridge-pulsar - sinkConfig: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - pulsar: - pulsarConfig: "test-pulsar-sink-config" - resources: - limits: - cpu: "0.2" - memory: 1.1G - requests: - cpu: "0.1" - memory: 1G - java: - jar: connectors/pulsar-io-aws-eventbridge-2.10.4.3.jar - clusterName: test-pulsar - autoAck: false -``` - -# How to use - -You can use the AWS EventBridge sink connector with Function Worker or Function Mesh. - -## Work with Function Worker - -{% callout title="Note" type="note" %} -Currently, the AWS EventBridge sink connector cannot run as a built-in connector as it uses the JAR package. -{% /callout %} - -1. Start a Pulsar cluster in standalone mode. - ``` - PULSAR_HOME/bin/pulsar standalone - ``` - -2. Run the AWS EventBridge sink connector. - ``` - PULSAR_HOME/bin/pulsar-admin sinks localrun \ - --sink-config-file - --archive - ``` - - Or, you can create a connector for the Pulsar cluster. - ``` - PULSAR_HOME/bin/pulsar-admin sinks create \ - --sink-config-file - --archive - ``` - -3. Send messages to a Pulsar topic. - - This example sends ten “hello” messages to the `test-aws-eventbridge-pulsar` topic in the `default` namespace of - the `public` tenant. - - ``` - PULSAR_HOME/bin/pulsar-client produce public/default/test-aws-eventbridge-pulsar --messages hello -n 10 - ``` - -4. Show data on AWS EventBridge. - -The connector will send the following format of JSON event to EventBridge. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": "hello" - } -} -``` - -You can configure the [rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-rules.html) to match events on -Amazon EventBridge, and set the target -to [Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html) to track -data. - -## Work with Function Mesh - -This example describes how to create an AWS EventBridge sink connector for a Kubernetes cluster using Function -Mesh. - -### Prerequisites - -- Create and connect to a [Kubernetes cluster](https://kubernetes.io/). - -- Create a [Pulsar cluster](https://pulsar.apache.org/docs/getting-started-helm/) in the Kubernetes cluster. - -- [Install the Function Mesh Operator and CRD](https://functionmesh.io/docs/install-function-mesh/) into the Kubernetes - cluster. - -- Prepare Amazon EventBridge service. For more details, - see [Getting Started with Amazon EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-setup.html). - -### Steps - -1. Define the AWS EventBridge sink connector with a YAML file and save it as `sink-sample.yaml`. - - This example shows how to publish the AWS EventBridge sink connector to Function Mesh with a Docker image. - ```yaml - apiVersion: compute.functionmesh.io/v1alpha1 - kind: Sink - metadata: - name: aws-eventbridge-sink-sample - spec: - image: streamnative/pulsar-io-aws-eventbridge:2.10.4.3 - replicas: 1 - maxReplicas: 1 - input: - topics: - - persistent://public/default/test-aws-eventbridge-pulsar - sinkConfig: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - pulsar: - pulsarConfig: "test-pulsar-sink-config" - resources: - limits: - cpu: "0.2" - memory: 1.1G - requests: - cpu: "0.1" - memory: 1G - java: - jar: connectors/pulsar-io-aws-eventbridge-2.10.4.3.jar - clusterName: test-pulsar - autoAck: false - ``` - -2. Apply the YAML file to create the AWS EventBridge sink connector. - - **Input** - - ``` - kubectl apply -f - ``` - - **Output** - - ``` - sink.compute.functionmesh.io/aws-eventbridge-sink-sample created - ``` - -3. Check whether the AWS EventBridge sink connector is created successfully. - - **Input** - - ``` - kubectl get all - ``` - - **Output** - - ``` - NAME READY STATUS RESTARTS AGE - pod/aws-eventbridge-sink-sample-0 1/1 Running 0 77s - ``` - - After that, you can produce and consume messages using the AWS EventBridge sink connector between Pulsar and - EventBridge. - - - diff --git a/connectors/aws-eventbridge-sink/v2.10.4.4/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v2.10.4.4/aws-eventbridge-sink.md deleted file mode 100644 index 4821a74b2..000000000 --- a/connectors/aws-eventbridge-sink/v2.10.4.4/aws-eventbridge-sink.md +++ /dev/null @@ -1,498 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: /images/connectors/aws-eventbridge.png -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: true -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v2.10.4.4/docs/aws-eventbridge-sink.png) - -# Features - -This section describes the features of the AWS EventBridge sink connector. For details about how to configure these -features, see [how to configure](#how-to-configure). - -## Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -> **Note:** -> Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the -> support of the Sink downstream system. - -## Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -## Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -## Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. -> - -## Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -## Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -# How to get - -This section describes how to get the AWS EventBridge sink connector. - -## Work with Function Worker - -You can get the AWS EventBridge sink connector using one of the following methods if you -use [Pulsar Function Worker](https://pulsar.apache.org/docs/en/functions-worker/) to run the connector in a cluster. - -- Download the NAR package - from [the download page](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases/). - -- Build it from the source code. - -To build the AWS EventBridge sink connector from the source code, follow these steps. - -1. Clone the source code to your machine. - - ```bash - git clone https://github.com/streamnative/pulsar-io-aws-eventbridge - ``` - -2. Build the connector in the `pulsar-io-aws-eventbridge` directory. - - ```bash - mvn clean install -DskipTests - ``` - - After the connector is successfully built, a `NAR` package is generated under the target directory. - - ```bash - ls target - pulsar-io-aws-eventbridge-2.10.4.4.nar - ``` - -## Work with Function Mesh - -You can pull the AWS EventBridge sink connector Docker image from -the [Docker Hub](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge) if you -use [Function Mesh](https://functionmesh.io/docs/connectors/run-connector) to run the connector. - -# How to configure - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | No | "" (empty string) | The EventBridge access key ID. | -| `secretAccessKey` | String | no | "" (empty string) | The EventBridge secret access key. | -| `role` | String | no | "" (empty string) | The AWS role to use. | -| `roleSessionName` | String | no | "" (empty string) | The AWS role session name to use. | -| `stsEndpoint` | String | no | "" (empty string) | The STS endpoint to use. By default, the [default STS endpoint](https://sts.amazonaws.com) is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | no | "" (empty string) | The STS region to use, By default, the 'region' config or env region is used. | -| `region` | String | yes | "" (empty string) | The EventBridge region. | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -## Work with Function Worker - -You can create a configuration file (JSON or YAML) to set the properties if you -use [Pulsar Function Worker](https://pulsar.apache.org/docs/en/functions-worker/) to run connectors in a cluster. - -**Example** - -* JSON - - ```json - { - "name": "eventbridge-sink", - "archive": "connectors/pulsar-io-aws-eventbridge-2.10.4.4.nar", - "tenant": "public", - "namespace": "default", - "inputs": [ - "test-aws-eventbridge-pulsar" - ], - "parallelism": 1, - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } - } - ``` - -* YAML - - ```yaml - name: eventbridge-sink - archive: 'connectors/pulsar-io-aws-eventbridge-2.10.4.4.nar' - tenant: public - namespace: default - inputs: - - test-aws-eventbridge-pulsar - parallelism: 1 - configs: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - ``` - -## Work with Function Mesh - -You can create -a [CustomResourceDefinitions (CRD)](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) -to create an AWS EventBridge sink connector. Using CRD makes Function Mesh naturally integrate with the Kubernetes -ecosystem. For more information about Pulsar sink CRD configurations, -see [sink CRD configurations](https://functionmesh.io/docs/connectors/io-crd-config/sink-crd-config). - -You can define a CRD file (YAML) to set the properties as below. - -```yaml -apiVersion: compute.functionmesh.io/v1alpha1 -kind: Sink -metadata: - name: aws-eventbridge-sink-sample -spec: - image: streamnative/pulsar-io-aws-eventbridge:2.10.4.4 - replicas: 1 - maxReplicas: 1 - input: - topics: - - persistent://public/default/test-aws-eventbridge-pulsar - sinkConfig: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - pulsar: - pulsarConfig: "test-pulsar-sink-config" - resources: - limits: - cpu: "0.2" - memory: 1.1G - requests: - cpu: "0.1" - memory: 1G - java: - jar: connectors/pulsar-io-aws-eventbridge-2.10.4.4.jar - clusterName: test-pulsar - autoAck: false -``` - -# How to use - -You can use the AWS EventBridge sink connector with Function Worker or Function Mesh. - -## Work with Function Worker - -> **Note** -> -> Currently, the AWS EventBridge sink connector cannot run as a built-in connector as it uses the JAR package. - -1. Start a Pulsar cluster in standalone mode. - ``` - PULSAR_HOME/bin/pulsar standalone - ``` - -2. Run the AWS EventBridge sink connector. - ``` - PULSAR_HOME/bin/pulsar-admin sinks localrun \ - --sink-config-file - --archive - ``` - - Or, you can create a connector for the Pulsar cluster. - ``` - PULSAR_HOME/bin/pulsar-admin sinks create \ - --sink-config-file - --archive - ``` - -3. Send messages to a Pulsar topic. - - This example sends ten “hello” messages to the `test-aws-eventbridge-pulsar` topic in the `default` namespace of - the `public` tenant. - - ``` - PULSAR_HOME/bin/pulsar-client produce public/default/test-aws-eventbridge-pulsar --messages hello -n 10 - ``` - -4. Show data on AWS EventBridge. - -The connector will send the following format of JSON event to EventBridge. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": "hello" - } -} -``` - -You can configure the [rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-rules.html) to match events on -Amazon EventBridge, and set the target -to [Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html) to track -data. - -## Work with Function Mesh - -This example describes how to create an AWS EventBridge sink connector for a Kubernetes cluster using Function -Mesh. - -### Prerequisites - -- Create and connect to a [Kubernetes cluster](https://kubernetes.io/). - -- Create a [Pulsar cluster](https://pulsar.apache.org/docs/getting-started-helm/) in the Kubernetes cluster. - -- [Install the Function Mesh Operator and CRD](https://functionmesh.io/docs/install-function-mesh/) into the Kubernetes - cluster. - -- Prepare Amazon EventBridge service. For more details, - see [Getting Started with Amazon EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-setup.html). - -### Steps - -1. Define the AWS EventBridge sink connector with a YAML file and save it as `sink-sample.yaml`. - - This example shows how to publish the AWS EventBridge sink connector to Function Mesh with a Docker image. - ```yaml - apiVersion: compute.functionmesh.io/v1alpha1 - kind: Sink - metadata: - name: aws-eventbridge-sink-sample - spec: - image: streamnative/pulsar-io-aws-eventbridge:2.10.4.4 - replicas: 1 - maxReplicas: 1 - input: - topics: - - persistent://public/default/test-aws-eventbridge-pulsar - sinkConfig: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - pulsar: - pulsarConfig: "test-pulsar-sink-config" - resources: - limits: - cpu: "0.2" - memory: 1.1G - requests: - cpu: "0.1" - memory: 1G - java: - jar: connectors/pulsar-io-aws-eventbridge-2.10.4.4.jar - clusterName: test-pulsar - autoAck: false - ``` - -2. Apply the YAML file to create the AWS EventBridge sink connector. - - **Input** - - ``` - kubectl apply -f - ``` - - **Output** - - ``` - sink.compute.functionmesh.io/aws-eventbridge-sink-sample created - ``` - -3. Check whether the AWS EventBridge sink connector is created successfully. - - **Input** - - ``` - kubectl get all - ``` - - **Output** - - ``` - NAME READY STATUS RESTARTS AGE - pod/aws-eventbridge-sink-sample-0 1/1 Running 0 77s - ``` - - After that, you can produce and consume messages using the AWS EventBridge sink connector between Pulsar and - EventBridge. - - - diff --git a/connectors/aws-eventbridge-sink/v2.10.4.5/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v2.10.4.5/aws-eventbridge-sink.md deleted file mode 100644 index 49b2bad6d..000000000 --- a/connectors/aws-eventbridge-sink/v2.10.4.5/aws-eventbridge-sink.md +++ /dev/null @@ -1,498 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: /images/connectors/aws-eventbridge.png -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: true -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v2.10.4.5/docs/aws-eventbridge-sink.png) - -# Features - -This section describes the features of the AWS EventBridge sink connector. For details about how to configure these -features, see [how to configure](#how-to-configure). - -## Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -> **Note:** -> Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the -> support of the Sink downstream system. - -## Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -## Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -## Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. -> - -## Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -## Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -# How to get - -This section describes how to get the AWS EventBridge sink connector. - -## Work with Function Worker - -You can get the AWS EventBridge sink connector using one of the following methods if you -use [Pulsar Function Worker](https://pulsar.apache.org/docs/en/functions-worker/) to run the connector in a cluster. - -- Download the NAR package - from [the download page](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases/). - -- Build it from the source code. - -To build the AWS EventBridge sink connector from the source code, follow these steps. - -1. Clone the source code to your machine. - - ```bash - git clone https://github.com/streamnative/pulsar-io-aws-eventbridge - ``` - -2. Build the connector in the `pulsar-io-aws-eventbridge` directory. - - ```bash - mvn clean install -DskipTests - ``` - - After the connector is successfully built, a `NAR` package is generated under the target directory. - - ```bash - ls target - pulsar-io-aws-eventbridge-2.10.4.5.nar - ``` - -## Work with Function Mesh - -You can pull the AWS EventBridge sink connector Docker image from -the [Docker Hub](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge) if you -use [Function Mesh](https://functionmesh.io/docs/connectors/run-connector) to run the connector. - -# How to configure - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | No | "" (empty string) | The EventBridge access key ID. | -| `secretAccessKey` | String | no | "" (empty string) | The EventBridge secret access key. | -| `role` | String | no | "" (empty string) | The AWS role to use. | -| `roleSessionName` | String | no | "" (empty string) | The AWS role session name to use. | -| `stsEndpoint` | String | no | "" (empty string) | The STS endpoint to use. By default, the [default STS endpoint](https://sts.amazonaws.com) is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | no | "" (empty string) | The STS region to use, By default, the 'region' config or env region is used. | -| `region` | String | yes | "" (empty string) | The EventBridge region. | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -## Work with Function Worker - -You can create a configuration file (JSON or YAML) to set the properties if you -use [Pulsar Function Worker](https://pulsar.apache.org/docs/en/functions-worker/) to run connectors in a cluster. - -**Example** - -* JSON - - ```json - { - "name": "eventbridge-sink", - "archive": "connectors/pulsar-io-aws-eventbridge-2.10.4.5.nar", - "tenant": "public", - "namespace": "default", - "inputs": [ - "test-aws-eventbridge-pulsar" - ], - "parallelism": 1, - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } - } - ``` - -* YAML - - ```yaml - name: eventbridge-sink - archive: 'connectors/pulsar-io-aws-eventbridge-2.10.4.5.nar' - tenant: public - namespace: default - inputs: - - test-aws-eventbridge-pulsar - parallelism: 1 - configs: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - ``` - -## Work with Function Mesh - -You can create -a [CustomResourceDefinitions (CRD)](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) -to create an AWS EventBridge sink connector. Using CRD makes Function Mesh naturally integrate with the Kubernetes -ecosystem. For more information about Pulsar sink CRD configurations, -see [sink CRD configurations](https://functionmesh.io/docs/connectors/io-crd-config/sink-crd-config). - -You can define a CRD file (YAML) to set the properties as below. - -```yaml -apiVersion: compute.functionmesh.io/v1alpha1 -kind: Sink -metadata: - name: aws-eventbridge-sink-sample -spec: - image: streamnative/pulsar-io-aws-eventbridge:2.10.4.5 - replicas: 1 - maxReplicas: 1 - input: - topics: - - persistent://public/default/test-aws-eventbridge-pulsar - sinkConfig: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - pulsar: - pulsarConfig: "test-pulsar-sink-config" - resources: - limits: - cpu: "0.2" - memory: 1.1G - requests: - cpu: "0.1" - memory: 1G - java: - jar: connectors/pulsar-io-aws-eventbridge-2.10.4.5.jar - clusterName: test-pulsar - autoAck: false -``` - -# How to use - -You can use the AWS EventBridge sink connector with Function Worker or Function Mesh. - -## Work with Function Worker - -> **Note** -> -> Currently, the AWS EventBridge sink connector cannot run as a built-in connector as it uses the JAR package. - -1. Start a Pulsar cluster in standalone mode. - ``` - PULSAR_HOME/bin/pulsar standalone - ``` - -2. Run the AWS EventBridge sink connector. - ``` - PULSAR_HOME/bin/pulsar-admin sinks localrun \ - --sink-config-file - --archive - ``` - - Or, you can create a connector for the Pulsar cluster. - ``` - PULSAR_HOME/bin/pulsar-admin sinks create \ - --sink-config-file - --archive - ``` - -3. Send messages to a Pulsar topic. - - This example sends ten “hello” messages to the `test-aws-eventbridge-pulsar` topic in the `default` namespace of - the `public` tenant. - - ``` - PULSAR_HOME/bin/pulsar-client produce public/default/test-aws-eventbridge-pulsar --messages hello -n 10 - ``` - -4. Show data on AWS EventBridge. - -The connector will send the following format of JSON event to EventBridge. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": "hello" - } -} -``` - -You can configure the [rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-rules.html) to match events on -Amazon EventBridge, and set the target -to [Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html) to track -data. - -## Work with Function Mesh - -This example describes how to create an AWS EventBridge sink connector for a Kubernetes cluster using Function -Mesh. - -### Prerequisites - -- Create and connect to a [Kubernetes cluster](https://kubernetes.io/). - -- Create a [Pulsar cluster](https://pulsar.apache.org/docs/getting-started-helm/) in the Kubernetes cluster. - -- [Install the Function Mesh Operator and CRD](https://functionmesh.io/docs/install-function-mesh/) into the Kubernetes - cluster. - -- Prepare Amazon EventBridge service. For more details, - see [Getting Started with Amazon EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-setup.html). - -### Steps - -1. Define the AWS EventBridge sink connector with a YAML file and save it as `sink-sample.yaml`. - - This example shows how to publish the AWS EventBridge sink connector to Function Mesh with a Docker image. - ```yaml - apiVersion: compute.functionmesh.io/v1alpha1 - kind: Sink - metadata: - name: aws-eventbridge-sink-sample - spec: - image: streamnative/pulsar-io-aws-eventbridge:2.10.4.5 - replicas: 1 - maxReplicas: 1 - input: - topics: - - persistent://public/default/test-aws-eventbridge-pulsar - sinkConfig: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - pulsar: - pulsarConfig: "test-pulsar-sink-config" - resources: - limits: - cpu: "0.2" - memory: 1.1G - requests: - cpu: "0.1" - memory: 1G - java: - jar: connectors/pulsar-io-aws-eventbridge-2.10.4.5.jar - clusterName: test-pulsar - autoAck: false - ``` - -2. Apply the YAML file to create the AWS EventBridge sink connector. - - **Input** - - ``` - kubectl apply -f - ``` - - **Output** - - ``` - sink.compute.functionmesh.io/aws-eventbridge-sink-sample created - ``` - -3. Check whether the AWS EventBridge sink connector is created successfully. - - **Input** - - ``` - kubectl get all - ``` - - **Output** - - ``` - NAME READY STATUS RESTARTS AGE - pod/aws-eventbridge-sink-sample-0 1/1 Running 0 77s - ``` - - After that, you can produce and consume messages using the AWS EventBridge sink connector between Pulsar and - EventBridge. - - - diff --git a/connectors/aws-eventbridge-sink/v2.10.4.6/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v2.10.4.6/aws-eventbridge-sink.md deleted file mode 100644 index 3f5ca14b0..000000000 --- a/connectors/aws-eventbridge-sink/v2.10.4.6/aws-eventbridge-sink.md +++ /dev/null @@ -1,542 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: /images/connectors/aws-eventbridge.png -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: true -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v2.10.4.6/docs/aws-eventbridge-sink.png) - -# Features - -This section describes the features of the AWS EventBridge sink connector. For details about how to configure these -features, see [how to configure](#how-to-configure). - -## Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -> **Note:** -> Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the -> support of the Sink downstream system. - -## Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -## Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -## Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. -> - -## Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -## Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -# How to get - -This section describes how to get the AWS EventBridge sink connector. - -## Prerequisites - -Prerequisites for an AWS EventBridge sink connector connecting to external systems: - -- Create [Access Keys](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) or [Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) -- Create EventBridge and EventBus in AWS. -- Create a rule in EventBridge. -- Ensure that the account has the following permissions to the AWS EventBus. For details, see [permissions for event buses](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-bus-perms.html) -```json - { - "Version": "2012-10-17", - "Statement": [ - - ### To grant permissions for an account to use the PutEvents action, include the following, otherwise delete this section: ### - - { - - "Sid": "AllowAccountToPutEvents", - "Effect": "Allow", - "Principal": { - "AWS": "" - }, - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - }, - - ### Include the following section to grant permissions to all members of your AWS Organizations to use the PutEvents action ### - - { - "Sid": "AllowAllAccountsFromOrganizationToPutEvents", - "Effect": "Allow", - "Principal": "*", - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - "Condition": { - "StringEquals": { - "aws:PrincipalOrgID": "o-yourOrgID" - } - } - } - ] -} -``` - -## Work with Function Worker - -You can get the AWS EventBridge sink connector using one of the following methods if you -use [Pulsar Function Worker](https://pulsar.apache.org/docs/en/functions-worker/) to run the connector in a cluster. - -- Download the NAR package - from [the download page](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases/). - -- Build it from the source code. - -To build the AWS EventBridge sink connector from the source code, follow these steps. - -1. Clone the source code to your machine. - - ```bash - git clone https://github.com/streamnative/pulsar-io-aws-eventbridge - ``` - -2. Build the connector in the `pulsar-io-aws-eventbridge` directory. - - ```bash - mvn clean install -DskipTests - ``` - - After the connector is successfully built, a `NAR` package is generated under the target directory. - - ```bash - ls target - pulsar-io-aws-eventbridge-2.10.4.6.nar - ``` - -## Work with Function Mesh - -You can pull the AWS EventBridge sink connector Docker image from -the [Docker Hub](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge) if you -use [Function Mesh](https://functionmesh.io/docs/connectors/run-connector) to run the connector. - -# How to configure - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | No | "" (empty string) | The EventBridge access key ID. | -| `secretAccessKey` | String | no | "" (empty string) | The EventBridge secret access key. | -| `role` | String | no | "" (empty string) | The AWS role to use. | -| `roleSessionName` | String | no | "" (empty string) | The AWS role session name to use. | -| `stsEndpoint` | String | no | "" (empty string) | The STS endpoint to use. By default, the [default STS endpoint](https://sts.amazonaws.com) is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | no | "" (empty string) | The STS region to use, By default, the 'region' config or env region is used. | -| `region` | String | yes | "" (empty string) | The EventBridge region. | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -## Work with Function Worker - -You can create a configuration file (JSON or YAML) to set the properties if you -use [Pulsar Function Worker](https://pulsar.apache.org/docs/en/functions-worker/) to run connectors in a cluster. - -**Example** - -* JSON - - ```json - { - "name": "eventbridge-sink", - "archive": "connectors/pulsar-io-aws-eventbridge-2.10.4.6.nar", - "tenant": "public", - "namespace": "default", - "inputs": [ - "test-aws-eventbridge-pulsar" - ], - "parallelism": 1, - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } - } - ``` - -* YAML - - ```yaml - name: eventbridge-sink - archive: 'connectors/pulsar-io-aws-eventbridge-2.10.4.6.nar' - tenant: public - namespace: default - inputs: - - test-aws-eventbridge-pulsar - parallelism: 1 - configs: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - ``` - -## Work with Function Mesh - -You can create -a [CustomResourceDefinitions (CRD)](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) -to create an AWS EventBridge sink connector. Using CRD makes Function Mesh naturally integrate with the Kubernetes -ecosystem. For more information about Pulsar sink CRD configurations, -see [sink CRD configurations](https://functionmesh.io/docs/connectors/io-crd-config/sink-crd-config). - -You can define a CRD file (YAML) to set the properties as below. - -```yaml -apiVersion: compute.functionmesh.io/v1alpha1 -kind: Sink -metadata: - name: aws-eventbridge-sink-sample -spec: - image: streamnative/pulsar-io-aws-eventbridge:2.10.4.6 - replicas: 1 - maxReplicas: 1 - input: - topics: - - persistent://public/default/test-aws-eventbridge-pulsar - sinkConfig: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - pulsar: - pulsarConfig: "test-pulsar-sink-config" - resources: - limits: - cpu: "0.2" - memory: 1.1G - requests: - cpu: "0.1" - memory: 1G - java: - jar: connectors/pulsar-io-aws-eventbridge-2.10.4.6.jar - clusterName: test-pulsar - autoAck: false -``` - -# How to use - -You can use the AWS EventBridge sink connector with Function Worker or Function Mesh. - -## Work with Function Worker - -> **Note** -> -> Currently, the AWS EventBridge sink connector cannot run as a built-in connector as it uses the JAR package. - -1. Start a Pulsar cluster in standalone mode. - ``` - PULSAR_HOME/bin/pulsar standalone - ``` - -2. Run the AWS EventBridge sink connector. - ``` - PULSAR_HOME/bin/pulsar-admin sinks localrun \ - --sink-config-file - --archive - ``` - - Or, you can create a connector for the Pulsar cluster. - ``` - PULSAR_HOME/bin/pulsar-admin sinks create \ - --sink-config-file - --archive - ``` - -3. Send messages to a Pulsar topic. - - This example sends ten “hello” messages to the `test-aws-eventbridge-pulsar` topic in the `default` namespace of - the `public` tenant. - - ``` - PULSAR_HOME/bin/pulsar-client produce public/default/test-aws-eventbridge-pulsar --messages hello -n 10 - ``` - -4. Show data on AWS EventBridge. - -The connector will send the following format of JSON event to EventBridge. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": "hello" - } -} -``` - -You can configure the [rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-rules.html) to match events on -Amazon EventBridge, and set the target -to [Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html) to track -data. - -## Work with Function Mesh - -This example describes how to create an AWS EventBridge sink connector for a Kubernetes cluster using Function -Mesh. - -### Prerequisites - -- Create and connect to a [Kubernetes cluster](https://kubernetes.io/). - -- Create a [Pulsar cluster](https://pulsar.apache.org/docs/getting-started-helm/) in the Kubernetes cluster. - -- [Install the Function Mesh Operator and CRD](https://functionmesh.io/docs/install-function-mesh/) into the Kubernetes - cluster. - -- Prepare Amazon EventBridge service. For more details, - see [Getting Started with Amazon EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-setup.html). - -### Steps - -1. Define the AWS EventBridge sink connector with a YAML file and save it as `sink-sample.yaml`. - - This example shows how to publish the AWS EventBridge sink connector to Function Mesh with a Docker image. - ```yaml - apiVersion: compute.functionmesh.io/v1alpha1 - kind: Sink - metadata: - name: aws-eventbridge-sink-sample - spec: - image: streamnative/pulsar-io-aws-eventbridge:2.10.4.6 - replicas: 1 - maxReplicas: 1 - input: - topics: - - persistent://public/default/test-aws-eventbridge-pulsar - sinkConfig: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - pulsar: - pulsarConfig: "test-pulsar-sink-config" - resources: - limits: - cpu: "0.2" - memory: 1.1G - requests: - cpu: "0.1" - memory: 1G - java: - jar: connectors/pulsar-io-aws-eventbridge-2.10.4.6.jar - clusterName: test-pulsar - autoAck: false - ``` - -2. Apply the YAML file to create the AWS EventBridge sink connector. - - **Input** - - ``` - kubectl apply -f - ``` - - **Output** - - ``` - sink.compute.functionmesh.io/aws-eventbridge-sink-sample created - ``` - -3. Check whether the AWS EventBridge sink connector is created successfully. - - **Input** - - ``` - kubectl get all - ``` - - **Output** - - ``` - NAME READY STATUS RESTARTS AGE - pod/aws-eventbridge-sink-sample-0 1/1 Running 0 77s - ``` - - After that, you can produce and consume messages using the AWS EventBridge sink connector between Pulsar and - EventBridge. - - - diff --git a/connectors/aws-eventbridge-sink/v2.10.4.7/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v2.10.4.7/aws-eventbridge-sink.md deleted file mode 100644 index 5585edfa6..000000000 --- a/connectors/aws-eventbridge-sink/v2.10.4.7/aws-eventbridge-sink.md +++ /dev/null @@ -1,542 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: /images/connectors/aws-eventbridge.png -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: true -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v2.10.4.7/docs/aws-eventbridge-sink.png) - -# Features - -This section describes the features of the AWS EventBridge sink connector. For details about how to configure these -features, see [how to configure](#how-to-configure). - -## Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -> **Note:** -> Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the -> support of the Sink downstream system. - -## Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -## Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -## Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. -> - -## Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -## Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -# How to get - -This section describes how to get the AWS EventBridge sink connector. - -## Prerequisites - -Prerequisites for an AWS EventBridge sink connector connecting to external systems: - -- Create [Access Keys](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) or [Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) -- Create EventBridge and EventBus in AWS. -- Create a rule in EventBridge. -- Ensure that the account has the following permissions to the AWS EventBus. For details, see [permissions for event buses](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-bus-perms.html) -```json - { - "Version": "2012-10-17", - "Statement": [ - - ### To grant permissions for an account to use the PutEvents action, include the following, otherwise delete this section: ### - - { - - "Sid": "AllowAccountToPutEvents", - "Effect": "Allow", - "Principal": { - "AWS": "" - }, - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - }, - - ### Include the following section to grant permissions to all members of your AWS Organizations to use the PutEvents action ### - - { - "Sid": "AllowAllAccountsFromOrganizationToPutEvents", - "Effect": "Allow", - "Principal": "*", - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - "Condition": { - "StringEquals": { - "aws:PrincipalOrgID": "o-yourOrgID" - } - } - } - ] -} -``` - -## Work with Function Worker - -You can get the AWS EventBridge sink connector using one of the following methods if you -use [Pulsar Function Worker](https://pulsar.apache.org/docs/en/functions-worker/) to run the connector in a cluster. - -- Download the NAR package - from [the download page](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases/). - -- Build it from the source code. - -To build the AWS EventBridge sink connector from the source code, follow these steps. - -1. Clone the source code to your machine. - - ```bash - git clone https://github.com/streamnative/pulsar-io-aws-eventbridge - ``` - -2. Build the connector in the `pulsar-io-aws-eventbridge` directory. - - ```bash - mvn clean install -DskipTests - ``` - - After the connector is successfully built, a `NAR` package is generated under the target directory. - - ```bash - ls target - pulsar-io-aws-eventbridge-2.10.4.7.nar - ``` - -## Work with Function Mesh - -You can pull the AWS EventBridge sink connector Docker image from -the [Docker Hub](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge) if you -use [Function Mesh](https://functionmesh.io/docs/connectors/run-connector) to run the connector. - -# How to configure - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | No | "" (empty string) | The EventBridge access key ID. | -| `secretAccessKey` | String | no | "" (empty string) | The EventBridge secret access key. | -| `role` | String | no | "" (empty string) | The AWS role to use. | -| `roleSessionName` | String | no | "" (empty string) | The AWS role session name to use. | -| `stsEndpoint` | String | no | "" (empty string) | The STS endpoint to use. By default, the [default STS endpoint](https://sts.amazonaws.com) is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | no | "" (empty string) | The STS region to use, By default, the 'region' config or env region is used. | -| `region` | String | yes | "" (empty string) | The EventBridge region. | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -## Work with Function Worker - -You can create a configuration file (JSON or YAML) to set the properties if you -use [Pulsar Function Worker](https://pulsar.apache.org/docs/en/functions-worker/) to run connectors in a cluster. - -**Example** - -* JSON - - ```json - { - "name": "eventbridge-sink", - "archive": "connectors/pulsar-io-aws-eventbridge-2.10.4.7.nar", - "tenant": "public", - "namespace": "default", - "inputs": [ - "test-aws-eventbridge-pulsar" - ], - "parallelism": 1, - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } - } - ``` - -* YAML - - ```yaml - name: eventbridge-sink - archive: 'connectors/pulsar-io-aws-eventbridge-2.10.4.7.nar' - tenant: public - namespace: default - inputs: - - test-aws-eventbridge-pulsar - parallelism: 1 - configs: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - ``` - -## Work with Function Mesh - -You can create -a [CustomResourceDefinitions (CRD)](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) -to create an AWS EventBridge sink connector. Using CRD makes Function Mesh naturally integrate with the Kubernetes -ecosystem. For more information about Pulsar sink CRD configurations, -see [sink CRD configurations](https://functionmesh.io/docs/connectors/io-crd-config/sink-crd-config). - -You can define a CRD file (YAML) to set the properties as below. - -```yaml -apiVersion: compute.functionmesh.io/v1alpha1 -kind: Sink -metadata: - name: aws-eventbridge-sink-sample -spec: - image: streamnative/pulsar-io-aws-eventbridge:2.10.4.7 - replicas: 1 - maxReplicas: 1 - input: - topics: - - persistent://public/default/test-aws-eventbridge-pulsar - sinkConfig: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - pulsar: - pulsarConfig: "test-pulsar-sink-config" - resources: - limits: - cpu: "0.2" - memory: 1.1G - requests: - cpu: "0.1" - memory: 1G - java: - jar: connectors/pulsar-io-aws-eventbridge-2.10.4.7.jar - clusterName: test-pulsar - autoAck: false -``` - -# How to use - -You can use the AWS EventBridge sink connector with Function Worker or Function Mesh. - -## Work with Function Worker - -> **Note** -> -> Currently, the AWS EventBridge sink connector cannot run as a built-in connector as it uses the JAR package. - -1. Start a Pulsar cluster in standalone mode. - ``` - PULSAR_HOME/bin/pulsar standalone - ``` - -2. Run the AWS EventBridge sink connector. - ``` - PULSAR_HOME/bin/pulsar-admin sinks localrun \ - --sink-config-file - --archive - ``` - - Or, you can create a connector for the Pulsar cluster. - ``` - PULSAR_HOME/bin/pulsar-admin sinks create \ - --sink-config-file - --archive - ``` - -3. Send messages to a Pulsar topic. - - This example sends ten “hello” messages to the `test-aws-eventbridge-pulsar` topic in the `default` namespace of - the `public` tenant. - - ``` - PULSAR_HOME/bin/pulsar-client produce public/default/test-aws-eventbridge-pulsar --messages hello -n 10 - ``` - -4. Show data on AWS EventBridge. - -The connector will send the following format of JSON event to EventBridge. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": "hello" - } -} -``` - -You can configure the [rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-rules.html) to match events on -Amazon EventBridge, and set the target -to [Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html) to track -data. - -## Work with Function Mesh - -This example describes how to create an AWS EventBridge sink connector for a Kubernetes cluster using Function -Mesh. - -### Prerequisites - -- Create and connect to a [Kubernetes cluster](https://kubernetes.io/). - -- Create a [Pulsar cluster](https://pulsar.apache.org/docs/getting-started-helm/) in the Kubernetes cluster. - -- [Install the Function Mesh Operator and CRD](https://functionmesh.io/docs/install-function-mesh/) into the Kubernetes - cluster. - -- Prepare Amazon EventBridge service. For more details, - see [Getting Started with Amazon EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-setup.html). - -### Steps - -1. Define the AWS EventBridge sink connector with a YAML file and save it as `sink-sample.yaml`. - - This example shows how to publish the AWS EventBridge sink connector to Function Mesh with a Docker image. - ```yaml - apiVersion: compute.functionmesh.io/v1alpha1 - kind: Sink - metadata: - name: aws-eventbridge-sink-sample - spec: - image: streamnative/pulsar-io-aws-eventbridge:2.10.4.7 - replicas: 1 - maxReplicas: 1 - input: - topics: - - persistent://public/default/test-aws-eventbridge-pulsar - sinkConfig: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - pulsar: - pulsarConfig: "test-pulsar-sink-config" - resources: - limits: - cpu: "0.2" - memory: 1.1G - requests: - cpu: "0.1" - memory: 1G - java: - jar: connectors/pulsar-io-aws-eventbridge-2.10.4.7.jar - clusterName: test-pulsar - autoAck: false - ``` - -2. Apply the YAML file to create the AWS EventBridge sink connector. - - **Input** - - ``` - kubectl apply -f - ``` - - **Output** - - ``` - sink.compute.functionmesh.io/aws-eventbridge-sink-sample created - ``` - -3. Check whether the AWS EventBridge sink connector is created successfully. - - **Input** - - ``` - kubectl get all - ``` - - **Output** - - ``` - NAME READY STATUS RESTARTS AGE - pod/aws-eventbridge-sink-sample-0 1/1 Running 0 77s - ``` - - After that, you can produce and consume messages using the AWS EventBridge sink connector between Pulsar and - EventBridge. - - - diff --git a/connectors/aws-eventbridge-sink/v2.10.4.8/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v2.10.4.8/aws-eventbridge-sink.md deleted file mode 100644 index 25d40d477..000000000 --- a/connectors/aws-eventbridge-sink/v2.10.4.8/aws-eventbridge-sink.md +++ /dev/null @@ -1,542 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: /images/connectors/aws-eventbridge.png -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: true -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v2.10.4.8/docs/aws-eventbridge-sink.png) - -# Features - -This section describes the features of the AWS EventBridge sink connector. For details about how to configure these -features, see [how to configure](#how-to-configure). - -## Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -> **Note:** -> Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the -> support of the Sink downstream system. - -## Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -## Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -## Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. -> - -## Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -## Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -# How to get - -This section describes how to get the AWS EventBridge sink connector. - -## Prerequisites - -Prerequisites for an AWS EventBridge sink connector connecting to external systems: - -- Create [Access Keys](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) or [Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) -- Create EventBridge and EventBus in AWS. -- Create a rule in EventBridge. -- Ensure that the account has the following permissions to the AWS EventBus. For details, see [permissions for event buses](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-bus-perms.html) -```json - { - "Version": "2012-10-17", - "Statement": [ - - ### To grant permissions for an account to use the PutEvents action, include the following, otherwise delete this section: ### - - { - - "Sid": "AllowAccountToPutEvents", - "Effect": "Allow", - "Principal": { - "AWS": "" - }, - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - }, - - ### Include the following section to grant permissions to all members of your AWS Organizations to use the PutEvents action ### - - { - "Sid": "AllowAllAccountsFromOrganizationToPutEvents", - "Effect": "Allow", - "Principal": "*", - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - "Condition": { - "StringEquals": { - "aws:PrincipalOrgID": "o-yourOrgID" - } - } - } - ] -} -``` - -## Work with Function Worker - -You can get the AWS EventBridge sink connector using one of the following methods if you -use [Pulsar Function Worker](https://pulsar.apache.org/docs/en/functions-worker/) to run the connector in a cluster. - -- Download the NAR package - from [the download page](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases/). - -- Build it from the source code. - -To build the AWS EventBridge sink connector from the source code, follow these steps. - -1. Clone the source code to your machine. - - ```bash - git clone https://github.com/streamnative/pulsar-io-aws-eventbridge - ``` - -2. Build the connector in the `pulsar-io-aws-eventbridge` directory. - - ```bash - mvn clean install -DskipTests - ``` - - After the connector is successfully built, a `NAR` package is generated under the target directory. - - ```bash - ls target - pulsar-io-aws-eventbridge-2.10.4.8.nar - ``` - -## Work with Function Mesh - -You can pull the AWS EventBridge sink connector Docker image from -the [Docker Hub](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge) if you -use [Function Mesh](https://functionmesh.io/docs/connectors/run-connector) to run the connector. - -# How to configure - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | No | "" (empty string) | The EventBridge access key ID. | -| `secretAccessKey` | String | no | "" (empty string) | The EventBridge secret access key. | -| `role` | String | no | "" (empty string) | The AWS role to use. | -| `roleSessionName` | String | no | "" (empty string) | The AWS role session name to use. | -| `stsEndpoint` | String | no | "" (empty string) | The STS endpoint to use. By default, the [default STS endpoint](https://sts.amazonaws.com) is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | no | "" (empty string) | The STS region to use, By default, the 'region' config or env region is used. | -| `region` | String | yes | "" (empty string) | The EventBridge region. | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -## Work with Function Worker - -You can create a configuration file (JSON or YAML) to set the properties if you -use [Pulsar Function Worker](https://pulsar.apache.org/docs/en/functions-worker/) to run connectors in a cluster. - -**Example** - -* JSON - - ```json - { - "name": "eventbridge-sink", - "archive": "connectors/pulsar-io-aws-eventbridge-2.10.4.8.nar", - "tenant": "public", - "namespace": "default", - "inputs": [ - "test-aws-eventbridge-pulsar" - ], - "parallelism": 1, - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } - } - ``` - -* YAML - - ```yaml - name: eventbridge-sink - archive: 'connectors/pulsar-io-aws-eventbridge-2.10.4.8.nar' - tenant: public - namespace: default - inputs: - - test-aws-eventbridge-pulsar - parallelism: 1 - configs: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - ``` - -## Work with Function Mesh - -You can create -a [CustomResourceDefinitions (CRD)](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) -to create an AWS EventBridge sink connector. Using CRD makes Function Mesh naturally integrate with the Kubernetes -ecosystem. For more information about Pulsar sink CRD configurations, -see [sink CRD configurations](https://functionmesh.io/docs/connectors/io-crd-config/sink-crd-config). - -You can define a CRD file (YAML) to set the properties as below. - -```yaml -apiVersion: compute.functionmesh.io/v1alpha1 -kind: Sink -metadata: - name: aws-eventbridge-sink-sample -spec: - image: streamnative/pulsar-io-aws-eventbridge:2.10.4.8 - replicas: 1 - maxReplicas: 1 - input: - topics: - - persistent://public/default/test-aws-eventbridge-pulsar - sinkConfig: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - pulsar: - pulsarConfig: "test-pulsar-sink-config" - resources: - limits: - cpu: "0.2" - memory: 1.1G - requests: - cpu: "0.1" - memory: 1G - java: - jar: connectors/pulsar-io-aws-eventbridge-2.10.4.8.jar - clusterName: test-pulsar - autoAck: false -``` - -# How to use - -You can use the AWS EventBridge sink connector with Function Worker or Function Mesh. - -## Work with Function Worker - -> **Note** -> -> Currently, the AWS EventBridge sink connector cannot run as a built-in connector as it uses the JAR package. - -1. Start a Pulsar cluster in standalone mode. - ``` - PULSAR_HOME/bin/pulsar standalone - ``` - -2. Run the AWS EventBridge sink connector. - ``` - PULSAR_HOME/bin/pulsar-admin sinks localrun \ - --sink-config-file - --archive - ``` - - Or, you can create a connector for the Pulsar cluster. - ``` - PULSAR_HOME/bin/pulsar-admin sinks create \ - --sink-config-file - --archive - ``` - -3. Send messages to a Pulsar topic. - - This example sends ten “hello” messages to the `test-aws-eventbridge-pulsar` topic in the `default` namespace of - the `public` tenant. - - ``` - PULSAR_HOME/bin/pulsar-client produce public/default/test-aws-eventbridge-pulsar --messages hello -n 10 - ``` - -4. Show data on AWS EventBridge. - -The connector will send the following format of JSON event to EventBridge. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": "hello" - } -} -``` - -You can configure the [rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-rules.html) to match events on -Amazon EventBridge, and set the target -to [Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html) to track -data. - -## Work with Function Mesh - -This example describes how to create an AWS EventBridge sink connector for a Kubernetes cluster using Function -Mesh. - -### Prerequisites - -- Create and connect to a [Kubernetes cluster](https://kubernetes.io/). - -- Create a [Pulsar cluster](https://pulsar.apache.org/docs/getting-started-helm/) in the Kubernetes cluster. - -- [Install the Function Mesh Operator and CRD](https://functionmesh.io/docs/install-function-mesh/) into the Kubernetes - cluster. - -- Prepare Amazon EventBridge service. For more details, - see [Getting Started with Amazon EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-setup.html). - -### Steps - -1. Define the AWS EventBridge sink connector with a YAML file and save it as `sink-sample.yaml`. - - This example shows how to publish the AWS EventBridge sink connector to Function Mesh with a Docker image. - ```yaml - apiVersion: compute.functionmesh.io/v1alpha1 - kind: Sink - metadata: - name: aws-eventbridge-sink-sample - spec: - image: streamnative/pulsar-io-aws-eventbridge:2.10.4.8 - replicas: 1 - maxReplicas: 1 - input: - topics: - - persistent://public/default/test-aws-eventbridge-pulsar - sinkConfig: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - pulsar: - pulsarConfig: "test-pulsar-sink-config" - resources: - limits: - cpu: "0.2" - memory: 1.1G - requests: - cpu: "0.1" - memory: 1G - java: - jar: connectors/pulsar-io-aws-eventbridge-2.10.4.8.jar - clusterName: test-pulsar - autoAck: false - ``` - -2. Apply the YAML file to create the AWS EventBridge sink connector. - - **Input** - - ``` - kubectl apply -f - ``` - - **Output** - - ``` - sink.compute.functionmesh.io/aws-eventbridge-sink-sample created - ``` - -3. Check whether the AWS EventBridge sink connector is created successfully. - - **Input** - - ``` - kubectl get all - ``` - - **Output** - - ``` - NAME READY STATUS RESTARTS AGE - pod/aws-eventbridge-sink-sample-0 1/1 Running 0 77s - ``` - - After that, you can produce and consume messages using the AWS EventBridge sink connector between Pulsar and - EventBridge. - - - diff --git a/connectors/aws-eventbridge-sink/v2.10.5.1/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v2.10.5.1/aws-eventbridge-sink.md deleted file mode 100644 index b1051052a..000000000 --- a/connectors/aws-eventbridge-sink/v2.10.5.1/aws-eventbridge-sink.md +++ /dev/null @@ -1,343 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: "/images/connectors/aws-eventbridge.png" -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: "true" -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v2.10.5.1/docs/aws-eventbridge-sink.png) - -## Quick start - -### Prerequisites - -The prerequisites for connecting an AWS EventBridge sink connector to external systems include: - -1. Create EventBridge and EventBus in AWS. -2. Create the [AWS User](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) and create `AccessKey`(Please record `AccessKey` and `AecretAccessKey`). -3. Assign permissions to Pulsar users, and ensure they have the following permissions to the AWS EventBus. For details, see [permissions for event buses](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-bus-perms.html) - -```json - { - "Version": "2012-10-17", - "Statement": [ - - ### To grant permissions for an account to use the PutEvents action, include the following, otherwise delete this section: ### - - { - - "Sid": "AllowAccountToPutEvents", - "Effect": "Allow", - "Principal": { - "AWS": "" - }, - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - }, - - ### Include the following section to grant permissions to all members of your AWS Organizations to use the PutEvents action ### - - { - "Sid": "AllowAllAccountsFromOrganizationToPutEvents", - "Effect": "Allow", - "Principal": "*", - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - "Condition": { - "StringEquals": { - "aws:PrincipalOrgID": "o-yourOrgID" - } - } - } - ] -} -``` -- You can set permissions directly for this user. With this method, when you create a connector, you only need to configure `accessKey` and `secretAccessKey`. -- Or you can use [Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html), this [video](https://www.youtube.com/watch?v=dqF4VJCska4) explains how to use STS on AWS. - With this method, when you create a connector, in addition to configuring `accessKey` and `secretAccessKey`, you also need to configure `role` and `roleSessionName`. - -4. Create a [Rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-rule.html) in EventBridge. -- The data structure sent to Event Bridge is described [here](## Metadata mapping), and you can create **event pattern** based on this structure. -- Set the target according to your needs. If you're testing this connector, you can set the target to [Cloud Watch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html). - - -### 1. Create a connector - -Depending on the environment, there are several ways to create an AWS EventBridge sink connector: - -- [Create Connector on StreamNative Cloud](https://docs.streamnative.io/docs/connector-create). -- [Create Connector with Function worker](https://pulsar.apache.org/docs/3.0.x/io-quickstart/). - Using this way requires you to download a **NAR** package to create a built-in or non-built-in connector. You can download the version you need from [here](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases). -- [Create Connector with Function mesh](https://functionmesh.io/docs/connectors/run-connector). - Using this way requires you to set the docker image. You can choose the version you want to launch from [here](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge/tags) - -No matter how you create an AWS EventBridge sink connector, the minimum configuration contains the following parameters. - -```json - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } -``` -> * The configuration structure varies depending on how you create the AWS EventBridge sink connector. -> For example, some are **JSON**, some are **YAML**, and some are **Kubernetes YAML**. You need to adapt the configs to the corresponding format. -> -> * If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. - -### 2. Send messages to the topic -> **Note:** If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information. - -``` java - PulsarClient client = PulsarClient.builder() - .serviceUrl("{{Your Pulsar URL}}") - .build(); - - Producer producer = client.newProducer(Schema.STRING) - .topic("{{Your topic name}}") - .create(); - - String message = "{\"msg\": \"msg-data\"}"; - MessageId msgID = producer.send(message); - System.out.println("Publish " + message + " and message ID " + msgID); - - producer.flush(); - producer.close(); - client.close(); -``` - -### 3. Show data on AWS EventBridge -The connector will send the following format of JSON event to EventBridge. -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail-type": "{{Your topic name}}", - "source": "{{Your connector name}}", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "msg": "msg-data" - }, - "message_id": "124:191:0" - } -} -``` - -## Configuration Properties - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | yes | "" (empty string) | The AWS EventBridge [access key ID.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `secretAccessKey` | String | yes | "" (empty string) | The AWS EventBridge [secret access key.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `region` | String | yes | "" (empty string) | The region where AWS EventBridge service is located. [All AWS region](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/Region.html) | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `role` | String | false | "" (empty string) | The AWS STS [roleArn](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). Example: arn:aws:iam::598203581484:role/test-role | -| `roleSessionName` | String | false | "" (empty string) | The AWS role session name, Name it yourself. | -| `stsEndpoint` | String | false | "" (empty string) | The AWS STS endpoint. By default, the default STS endpoint: https://sts.amazonaws.com is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | false | "" (empty string) | The AWS STS region, By default, the 'region' config or env region is used. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). Example: `arn:aws:events:ap-northeast-1:598263551484:event-bus/my_eventbus` | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -> For details about this connector's advanced features and configurations, see [Advanced features](#advanced-features). - -## Advanced features - -### Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -> **Note:** -> Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the -> support of the Sink downstream system. - -### Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -### Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -### Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. - -### Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -### Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -## More Links -- [GitHub Repo](https://github.com/streamnative/pulsar-io-aws-eventbridge) -- [Announcing the Amazon EventBridge Sink Connector for Apache Pulsar](https://streamnative.io/blog/announcing-the-amazon-eventbridge-sink-connector-for-apache-pulsar) -- [Amazon EventBridge connector is now integrated with StreamNative Cloud](https://streamnative.io/blog/amazon-eventbridge-connector-is-now-integrated-with-streamnative-cloud) - - diff --git a/connectors/aws-eventbridge-sink/v2.10.5.2/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v2.10.5.2/aws-eventbridge-sink.md deleted file mode 100644 index 68ac4ac47..000000000 --- a/connectors/aws-eventbridge-sink/v2.10.5.2/aws-eventbridge-sink.md +++ /dev/null @@ -1,346 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: "/images/connectors/aws-eventbridge.png" -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: "true" -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v2.10.5.2/docs/aws-eventbridge-sink.png) - -## Quick start - -### Prerequisites - -The prerequisites for connecting an AWS EventBridge sink connector to external systems include: - -1. Create EventBridge and EventBus in AWS. -2. Create the [AWS User](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) and create `AccessKey`(Please record `AccessKey` and `SecretAccessKey`). -3. Assign permissions to AWS User, and ensure they have the following permissions to the AWS EventBus. For details, see [permissions for event buses](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-bus-perms.html) - -```json - { - "Version": "2012-10-17", - "Statement": [ - - ### To grant permissions for an account to use the PutEvents action, include the following, otherwise delete this section: ### - - { - - "Sid": "AllowAccountToPutEvents", - "Effect": "Allow", - "Principal": { - "AWS": "" - }, - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - }, - - ### Include the following section to grant permissions to all members of your AWS Organizations to use the PutEvents action ### - - { - "Sid": "AllowAllAccountsFromOrganizationToPutEvents", - "Effect": "Allow", - "Principal": "*", - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - "Condition": { - "StringEquals": { - "aws:PrincipalOrgID": "o-yourOrgID" - } - } - } - ] -} -``` -- You can set permissions directly for this user. With this method, when you create a connector, you only need to configure `accessKey` and `secretAccessKey`. -- Or you can use [Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html), this [video](https://www.youtube.com/watch?v=dqF4VJCska4) explains how to use STS on AWS. - With this method, when you create a connector, in addition to configuring `accessKey` and `secretAccessKey`, you also need to configure `role` and `roleSessionName`. - -4. Create a [Rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-rule.html) in EventBridge. -- The data structure sent to Event Bridge is described [here](## Metadata mapping), and you can create **event pattern** based on this structure. -- Set the target according to your needs. If you're testing this connector, you can set the target to [Cloud Watch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html). - - -### 1. Create a connector - -Depending on the environment, there are several ways to create an AWS EventBridge sink connector: - -- [Create Connector on StreamNative Cloud](https://docs.streamnative.io/docs/connector-create). -- [Create Connector with Function worker](https://pulsar.apache.org/docs/3.0.x/io-quickstart/). - Using this way requires you to download a **NAR** package to create a built-in or non-built-in connector. You can download the version you need from [here](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases). -- [Create Connector with Function mesh](https://functionmesh.io/docs/connectors/run-connector). - Using this way requires you to set the docker image. You can choose the version you want to launch from [here](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge/tags) - -No matter how you create an AWS EventBridge sink connector, the minimum configuration contains the following parameters. - -```json - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } -``` -> * The configuration structure varies depending on how you create the AWS EventBridge sink connector. -> For example, some are **JSON**, some are **YAML**, and some are **Kubernetes YAML**. You need to adapt the configs to the corresponding format. -> -> * If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. - -### 2. Send messages to the topic - -{% callout title="Note" type="note" %} -If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information. -{% /callout %} - -``` java - PulsarClient client = PulsarClient.builder() - .serviceUrl("{{Your Pulsar URL}}") - .build(); - - Producer producer = client.newProducer(Schema.STRING) - .topic("{{Your topic name}}") - .create(); - - String message = "{\"msg\": \"msg-data\"}"; - MessageId msgID = producer.send(message); - System.out.println("Publish " + message + " and message ID " + msgID); - - producer.flush(); - producer.close(); - client.close(); -``` - -### 3. Show data on AWS EventBridge -The connector will send the following format of JSON event to EventBridge. -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail-type": "{{Your topic name}}", - "source": "{{Your connector name}}", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "msg": "msg-data" - }, - "message_id": "124:191:0" - } -} -``` - -## Configuration Properties - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | yes | "" (empty string) | The AWS EventBridge [access key ID.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `secretAccessKey` | String | yes | "" (empty string) | The AWS EventBridge [secret access key.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `region` | String | yes | "" (empty string) | The region where AWS EventBridge service is located. [All AWS region](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/Region.html) | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `role` | String | false | "" (empty string) | The AWS STS [roleArn](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). Example: arn:aws:iam::598203581484:role/test-role | -| `roleSessionName` | String | false | "" (empty string) | The AWS role session name, Name it yourself. | -| `stsEndpoint` | String | false | "" (empty string) | The AWS STS endpoint. By default, the default STS endpoint: https://sts.amazonaws.com is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | false | "" (empty string) | The AWS STS region, By default, the 'region' config or env region is used. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). Example: `arn:aws:events:ap-northeast-1:598263551484:event-bus/my_eventbus` | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -> For details about this connector's advanced features and configurations, see [Advanced features](#advanced-features). - -## Advanced features - -### Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -{% callout title="Note" type="note" %} -Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the support of the Sink downstream system. -{% /callout %} - -### Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -### Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -### Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. - -### Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -### Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -## More Links -- [GitHub Repo](https://github.com/streamnative/pulsar-io-aws-eventbridge) -- [Announcing the Amazon EventBridge Sink Connector for Apache Pulsar](https://streamnative.io/blog/announcing-the-amazon-eventbridge-sink-connector-for-apache-pulsar) -- [Amazon EventBridge connector is now integrated with StreamNative Cloud](https://streamnative.io/blog/amazon-eventbridge-connector-is-now-integrated-with-streamnative-cloud) - - diff --git a/connectors/aws-eventbridge-sink/v2.10.5.3/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v2.10.5.3/aws-eventbridge-sink.md deleted file mode 100644 index 33dccd919..000000000 --- a/connectors/aws-eventbridge-sink/v2.10.5.3/aws-eventbridge-sink.md +++ /dev/null @@ -1,346 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: "/images/connectors/aws-eventbridge.png" -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: "true" -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v2.10.5.3/docs/aws-eventbridge-sink.png) - -## Quick start - -### Prerequisites - -The prerequisites for connecting an AWS EventBridge sink connector to external systems include: - -1. Create EventBridge and EventBus in AWS. -2. Create the [AWS User](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) and create `AccessKey`(Please record `AccessKey` and `SecretAccessKey`). -3. Assign permissions to AWS User, and ensure they have the following permissions to the AWS EventBus. For details, see [permissions for event buses](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-bus-perms.html) - -```json - { - "Version": "2012-10-17", - "Statement": [ - - ### To grant permissions for an account to use the PutEvents action, include the following, otherwise delete this section: ### - - { - - "Sid": "AllowAccountToPutEvents", - "Effect": "Allow", - "Principal": { - "AWS": "" - }, - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - }, - - ### Include the following section to grant permissions to all members of your AWS Organizations to use the PutEvents action ### - - { - "Sid": "AllowAllAccountsFromOrganizationToPutEvents", - "Effect": "Allow", - "Principal": "*", - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - "Condition": { - "StringEquals": { - "aws:PrincipalOrgID": "o-yourOrgID" - } - } - } - ] -} -``` -- You can set permissions directly for this user. With this method, when you create a connector, you only need to configure `accessKey` and `secretAccessKey`. -- Or you can use [Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html), this [video](https://www.youtube.com/watch?v=dqF4VJCska4) explains how to use STS on AWS. - With this method, when you create a connector, in addition to configuring `accessKey` and `secretAccessKey`, you also need to configure `role` and `roleSessionName`. - -4. Create a [Rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-rule.html) in EventBridge. -- The data structure sent to Event Bridge is described [here](## Metadata mapping), and you can create **event pattern** based on this structure. -- Set the target according to your needs. If you're testing this connector, you can set the target to [Cloud Watch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html). - - -### 1. Create a connector - -Depending on the environment, there are several ways to create an AWS EventBridge sink connector: - -- [Create Connector on StreamNative Cloud](https://docs.streamnative.io/docs/connector-create). -- [Create Connector with Function worker](https://pulsar.apache.org/docs/3.0.x/io-quickstart/). - Using this way requires you to download a **NAR** package to create a built-in or non-built-in connector. You can download the version you need from [here](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases). -- [Create Connector with Function mesh](https://functionmesh.io/docs/connectors/run-connector). - Using this way requires you to set the docker image. You can choose the version you want to launch from [here](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge/tags) - -No matter how you create an AWS EventBridge sink connector, the minimum configuration contains the following parameters. - -```json - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } -``` -> * The configuration structure varies depending on how you create the AWS EventBridge sink connector. -> For example, some are **JSON**, some are **YAML**, and some are **Kubernetes YAML**. You need to adapt the configs to the corresponding format. -> -> * If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. - -### 2. Send messages to the topic - -{% callout title="Note" type="note" %} -If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information. -{% /callout %} - -``` java - PulsarClient client = PulsarClient.builder() - .serviceUrl("{{Your Pulsar URL}}") - .build(); - - Producer producer = client.newProducer(Schema.STRING) - .topic("{{Your topic name}}") - .create(); - - String message = "{\"msg\": \"msg-data\"}"; - MessageId msgID = producer.send(message); - System.out.println("Publish " + message + " and message ID " + msgID); - - producer.flush(); - producer.close(); - client.close(); -``` - -### 3. Show data on AWS EventBridge -The connector will send the following format of JSON event to EventBridge. -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail-type": "{{Your topic name}}", - "source": "{{Your connector name}}", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "msg": "msg-data" - }, - "message_id": "124:191:0" - } -} -``` - -## Configuration Properties - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | yes | "" (empty string) | The AWS EventBridge [access key ID.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `secretAccessKey` | String | yes | "" (empty string) | The AWS EventBridge [secret access key.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `region` | String | yes | "" (empty string) | The region where AWS EventBridge service is located. [All AWS region](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/Region.html) | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `role` | String | false | "" (empty string) | The AWS STS [roleArn](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). Example: arn:aws:iam::598203581484:role/test-role | -| `roleSessionName` | String | false | "" (empty string) | The AWS role session name, Name it yourself. | -| `stsEndpoint` | String | false | "" (empty string) | The AWS STS endpoint. By default, the default STS endpoint: https://sts.amazonaws.com is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | false | "" (empty string) | The AWS STS region, By default, the 'region' config or env region is used. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). Example: `arn:aws:events:ap-northeast-1:598263551484:event-bus/my_eventbus` | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -> For details about this connector's advanced features and configurations, see [Advanced features](#advanced-features). - -## Advanced features - -### Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -{% callout title="Note" type="note" %} -Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the support of the Sink downstream system. -{% /callout %} - -### Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -### Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -### Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. - -### Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -### Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -## More Links -- [GitHub Repo](https://github.com/streamnative/pulsar-io-aws-eventbridge) -- [Announcing the Amazon EventBridge Sink Connector for Apache Pulsar](https://streamnative.io/blog/announcing-the-amazon-eventbridge-sink-connector-for-apache-pulsar) -- [Amazon EventBridge connector is now integrated with StreamNative Cloud](https://streamnative.io/blog/amazon-eventbridge-connector-is-now-integrated-with-streamnative-cloud) - - diff --git a/connectors/aws-eventbridge-sink/v2.10.5.4/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v2.10.5.4/aws-eventbridge-sink.md deleted file mode 100644 index 09f310776..000000000 --- a/connectors/aws-eventbridge-sink/v2.10.5.4/aws-eventbridge-sink.md +++ /dev/null @@ -1,346 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: "/images/connectors/aws-eventbridge.png" -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: "true" -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v2.10.5.4/docs/aws-eventbridge-sink.png) - -## Quick start - -### Prerequisites - -The prerequisites for connecting an AWS EventBridge sink connector to external systems include: - -1. Create EventBridge and EventBus in AWS. -2. Create the [AWS User](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) and create `AccessKey`(Please record `AccessKey` and `SecretAccessKey`). -3. Assign permissions to AWS User, and ensure they have the following permissions to the AWS EventBus. For details, see [permissions for event buses](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-bus-perms.html) - -```json - { - "Version": "2012-10-17", - "Statement": [ - - ### To grant permissions for an account to use the PutEvents action, include the following, otherwise delete this section: ### - - { - - "Sid": "AllowAccountToPutEvents", - "Effect": "Allow", - "Principal": { - "AWS": "" - }, - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - }, - - ### Include the following section to grant permissions to all members of your AWS Organizations to use the PutEvents action ### - - { - "Sid": "AllowAllAccountsFromOrganizationToPutEvents", - "Effect": "Allow", - "Principal": "*", - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - "Condition": { - "StringEquals": { - "aws:PrincipalOrgID": "o-yourOrgID" - } - } - } - ] -} -``` -- You can set permissions directly for this user. With this method, when you create a connector, you only need to configure `accessKey` and `secretAccessKey`. -- Or you can use [Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html), this [video](https://www.youtube.com/watch?v=dqF4VJCska4) explains how to use STS on AWS. - With this method, when you create a connector, in addition to configuring `accessKey` and `secretAccessKey`, you also need to configure `role` and `roleSessionName`. - -4. Create a [Rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-rule.html) in EventBridge. -- The data structure sent to Event Bridge is described [here](## Metadata mapping), and you can create **event pattern** based on this structure. -- Set the target according to your needs. If you're testing this connector, you can set the target to [Cloud Watch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html). - - -### 1. Create a connector - -Depending on the environment, there are several ways to create an AWS EventBridge sink connector: - -- [Create Connector on StreamNative Cloud](https://docs.streamnative.io/docs/connector-create). -- [Create Connector with Function worker](https://pulsar.apache.org/docs/3.0.x/io-quickstart/). - Using this way requires you to download a **NAR** package to create a built-in or non-built-in connector. You can download the version you need from [here](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases). -- [Create Connector with Function mesh](https://functionmesh.io/docs/connectors/run-connector). - Using this way requires you to set the docker image. You can choose the version you want to launch from [here](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge/tags) - -No matter how you create an AWS EventBridge sink connector, the minimum configuration contains the following parameters. - -```json - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } -``` -> * The configuration structure varies depending on how you create the AWS EventBridge sink connector. -> For example, some are **JSON**, some are **YAML**, and some are **Kubernetes YAML**. You need to adapt the configs to the corresponding format. -> -> * If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. - -### 2. Send messages to the topic - -{% callout title="Note" type="note" %} -If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information. -{% /callout %} - -``` java - PulsarClient client = PulsarClient.builder() - .serviceUrl("{{Your Pulsar URL}}") - .build(); - - Producer producer = client.newProducer(Schema.STRING) - .topic("{{Your topic name}}") - .create(); - - String message = "{\"msg\": \"msg-data\"}"; - MessageId msgID = producer.send(message); - System.out.println("Publish " + message + " and message ID " + msgID); - - producer.flush(); - producer.close(); - client.close(); -``` - -### 3. Show data on AWS EventBridge -The connector will send the following format of JSON event to EventBridge. -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail-type": "{{Your topic name}}", - "source": "{{Your connector name}}", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "msg": "msg-data" - }, - "message_id": "124:191:0" - } -} -``` - -## Configuration Properties - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | yes | "" (empty string) | The AWS EventBridge [access key ID.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `secretAccessKey` | String | yes | "" (empty string) | The AWS EventBridge [secret access key.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `region` | String | yes | "" (empty string) | The region where AWS EventBridge service is located. [All AWS region](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/Region.html) | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `role` | String | false | "" (empty string) | The AWS STS [roleArn](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). Example: arn:aws:iam::598203581484:role/test-role | -| `roleSessionName` | String | false | "" (empty string) | The AWS role session name, Name it yourself. | -| `stsEndpoint` | String | false | "" (empty string) | The AWS STS endpoint. By default, the default STS endpoint: https://sts.amazonaws.com is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | false | "" (empty string) | The AWS STS region, By default, the 'region' config or env region is used. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). Example: `arn:aws:events:ap-northeast-1:598263551484:event-bus/my_eventbus` | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -> For details about this connector's advanced features and configurations, see [Advanced features](#advanced-features). - -## Advanced features - -### Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -{% callout title="Note" type="note" %} -Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the support of the Sink downstream system. -{% /callout %} - -### Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -### Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -### Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. - -### Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -### Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -## More Links -- [GitHub Repo](https://github.com/streamnative/pulsar-io-aws-eventbridge) -- [Announcing the Amazon EventBridge Sink Connector for Apache Pulsar](https://streamnative.io/blog/announcing-the-amazon-eventbridge-sink-connector-for-apache-pulsar) -- [Amazon EventBridge connector is now integrated with StreamNative Cloud](https://streamnative.io/blog/amazon-eventbridge-connector-is-now-integrated-with-streamnative-cloud) - - diff --git a/connectors/aws-eventbridge-sink/v2.10.5.5/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v2.10.5.5/aws-eventbridge-sink.md deleted file mode 100644 index 76b05873d..000000000 --- a/connectors/aws-eventbridge-sink/v2.10.5.5/aws-eventbridge-sink.md +++ /dev/null @@ -1,346 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: "/images/connectors/aws-eventbridge.png" -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: "true" -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v2.10.5.5/docs/aws-eventbridge-sink.png) - -## Quick start - -### Prerequisites - -The prerequisites for connecting an AWS EventBridge sink connector to external systems include: - -1. Create EventBridge and EventBus in AWS. -2. Create the [AWS User](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) and create `AccessKey`(Please record `AccessKey` and `SecretAccessKey`). -3. Assign permissions to AWS User, and ensure they have the following permissions to the AWS EventBus. For details, see [permissions for event buses](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-bus-perms.html) - -```json - { - "Version": "2012-10-17", - "Statement": [ - - ### To grant permissions for an account to use the PutEvents action, include the following, otherwise delete this section: ### - - { - - "Sid": "AllowAccountToPutEvents", - "Effect": "Allow", - "Principal": { - "AWS": "" - }, - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - }, - - ### Include the following section to grant permissions to all members of your AWS Organizations to use the PutEvents action ### - - { - "Sid": "AllowAllAccountsFromOrganizationToPutEvents", - "Effect": "Allow", - "Principal": "*", - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - "Condition": { - "StringEquals": { - "aws:PrincipalOrgID": "o-yourOrgID" - } - } - } - ] -} -``` -- You can set permissions directly for this user. With this method, when you create a connector, you only need to configure `accessKey` and `secretAccessKey`. -- Or you can use [Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html), this [video](https://www.youtube.com/watch?v=dqF4VJCska4) explains how to use STS on AWS. - With this method, when you create a connector, in addition to configuring `accessKey` and `secretAccessKey`, you also need to configure `role` and `roleSessionName`. - -4. Create a [Rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-rule.html) in EventBridge. -- The data structure sent to Event Bridge is described [here](## Metadata mapping), and you can create **event pattern** based on this structure. -- Set the target according to your needs. If you're testing this connector, you can set the target to [Cloud Watch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html). - - -### 1. Create a connector - -Depending on the environment, there are several ways to create an AWS EventBridge sink connector: - -- [Create Connector on StreamNative Cloud](https://docs.streamnative.io/docs/connector-create). -- [Create Connector with Function worker](https://pulsar.apache.org/docs/3.0.x/io-quickstart/). - Using this way requires you to download a **NAR** package to create a built-in or non-built-in connector. You can download the version you need from [here](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases). -- [Create Connector with Function mesh](https://functionmesh.io/docs/connectors/run-connector). - Using this way requires you to set the docker image. You can choose the version you want to launch from [here](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge/tags) - -No matter how you create an AWS EventBridge sink connector, the minimum configuration contains the following parameters. - -```json - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } -``` -> * The configuration structure varies depending on how you create the AWS EventBridge sink connector. -> For example, some are **JSON**, some are **YAML**, and some are **Kubernetes YAML**. You need to adapt the configs to the corresponding format. -> -> * If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. - -### 2. Send messages to the topic - -{% callout title="Note" type="note" %} -If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information. -{% /callout %} - -``` java - PulsarClient client = PulsarClient.builder() - .serviceUrl("{{Your Pulsar URL}}") - .build(); - - Producer producer = client.newProducer(Schema.STRING) - .topic("{{Your topic name}}") - .create(); - - String message = "{\"msg\": \"msg-data\"}"; - MessageId msgID = producer.send(message); - System.out.println("Publish " + message + " and message ID " + msgID); - - producer.flush(); - producer.close(); - client.close(); -``` - -### 3. Show data on AWS EventBridge -The connector will send the following format of JSON event to EventBridge. -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail-type": "{{Your topic name}}", - "source": "{{Your connector name}}", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "msg": "msg-data" - }, - "message_id": "124:191:0" - } -} -``` - -## Configuration Properties - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | yes | "" (empty string) | The AWS EventBridge [access key ID.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `secretAccessKey` | String | yes | "" (empty string) | The AWS EventBridge [secret access key.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `region` | String | yes | "" (empty string) | The region where AWS EventBridge service is located. [All AWS region](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/Region.html) | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `role` | String | false | "" (empty string) | The AWS STS [roleArn](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). Example: arn:aws:iam::598203581484:role/test-role | -| `roleSessionName` | String | false | "" (empty string) | The AWS role session name, Name it yourself. | -| `stsEndpoint` | String | false | "" (empty string) | The AWS STS endpoint. By default, the default STS endpoint: https://sts.amazonaws.com is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | false | "" (empty string) | The AWS STS region, By default, the 'region' config or env region is used. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). Example: `arn:aws:events:ap-northeast-1:598263551484:event-bus/my_eventbus` | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -> For details about this connector's advanced features and configurations, see [Advanced features](#advanced-features). - -## Advanced features - -### Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -{% callout title="Note" type="note" %} -Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the support of the Sink downstream system. -{% /callout %} - -### Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -### Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -### Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. - -### Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -### Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -## More Links -- [GitHub Repo](https://github.com/streamnative/pulsar-io-aws-eventbridge) -- [Announcing the Amazon EventBridge Sink Connector for Apache Pulsar](https://streamnative.io/blog/announcing-the-amazon-eventbridge-sink-connector-for-apache-pulsar) -- [Amazon EventBridge connector is now integrated with StreamNative Cloud](https://streamnative.io/blog/amazon-eventbridge-connector-is-now-integrated-with-streamnative-cloud) - - diff --git a/connectors/aws-eventbridge-sink/v2.10.5.6/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v2.10.5.6/aws-eventbridge-sink.md deleted file mode 100644 index 8805f5584..000000000 --- a/connectors/aws-eventbridge-sink/v2.10.5.6/aws-eventbridge-sink.md +++ /dev/null @@ -1,346 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: "/images/connectors/aws-eventbridge.png" -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: "true" -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v2.10.5.6/docs/aws-eventbridge-sink.png) - -## Quick start - -### Prerequisites - -The prerequisites for connecting an AWS EventBridge sink connector to external systems include: - -1. Create EventBridge and EventBus in AWS. -2. Create the [AWS User](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) and create `AccessKey`(Please record `AccessKey` and `SecretAccessKey`). -3. Assign permissions to AWS User, and ensure they have the following permissions to the AWS EventBus. For details, see [permissions for event buses](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-bus-perms.html) - -```json - { - "Version": "2012-10-17", - "Statement": [ - - ### To grant permissions for an account to use the PutEvents action, include the following, otherwise delete this section: ### - - { - - "Sid": "AllowAccountToPutEvents", - "Effect": "Allow", - "Principal": { - "AWS": "" - }, - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - }, - - ### Include the following section to grant permissions to all members of your AWS Organizations to use the PutEvents action ### - - { - "Sid": "AllowAllAccountsFromOrganizationToPutEvents", - "Effect": "Allow", - "Principal": "*", - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - "Condition": { - "StringEquals": { - "aws:PrincipalOrgID": "o-yourOrgID" - } - } - } - ] -} -``` -- You can set permissions directly for this user. With this method, when you create a connector, you only need to configure `accessKey` and `secretAccessKey`. -- Or you can use [Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html), this [video](https://www.youtube.com/watch?v=dqF4VJCska4) explains how to use STS on AWS. - With this method, when you create a connector, in addition to configuring `accessKey` and `secretAccessKey`, you also need to configure `role` and `roleSessionName`. - -4. Create a [Rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-rule.html) in EventBridge. -- The data structure sent to Event Bridge is described [here](## Metadata mapping), and you can create **event pattern** based on this structure. -- Set the target according to your needs. If you're testing this connector, you can set the target to [Cloud Watch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html). - - -### 1. Create a connector - -Depending on the environment, there are several ways to create an AWS EventBridge sink connector: - -- [Create Connector on StreamNative Cloud](https://docs.streamnative.io/docs/connector-create). -- [Create Connector with Function worker](https://pulsar.apache.org/docs/3.0.x/io-quickstart/). - Using this way requires you to download a **NAR** package to create a built-in or non-built-in connector. You can download the version you need from [here](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases). -- [Create Connector with Function mesh](https://functionmesh.io/docs/connectors/run-connector). - Using this way requires you to set the docker image. You can choose the version you want to launch from [here](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge/tags) - -No matter how you create an AWS EventBridge sink connector, the minimum configuration contains the following parameters. - -```json - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } -``` -> * The configuration structure varies depending on how you create the AWS EventBridge sink connector. -> For example, some are **JSON**, some are **YAML**, and some are **Kubernetes YAML**. You need to adapt the configs to the corresponding format. -> -> * If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. - -### 2. Send messages to the topic - -{% callout title="Note" type="note" %} -If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information. -{% /callout %} - -``` java - PulsarClient client = PulsarClient.builder() - .serviceUrl("{{Your Pulsar URL}}") - .build(); - - Producer producer = client.newProducer(Schema.STRING) - .topic("{{Your topic name}}") - .create(); - - String message = "{\"msg\": \"msg-data\"}"; - MessageId msgID = producer.send(message); - System.out.println("Publish " + message + " and message ID " + msgID); - - producer.flush(); - producer.close(); - client.close(); -``` - -### 3. Show data on AWS EventBridge -The connector will send the following format of JSON event to EventBridge. -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail-type": "{{Your topic name}}", - "source": "{{Your connector name}}", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "msg": "msg-data" - }, - "message_id": "124:191:0" - } -} -``` - -## Configuration Properties - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | yes | "" (empty string) | The AWS EventBridge [access key ID.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `secretAccessKey` | String | yes | "" (empty string) | The AWS EventBridge [secret access key.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `region` | String | yes | "" (empty string) | The region where AWS EventBridge service is located. [All AWS region](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/Region.html) | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `role` | String | false | "" (empty string) | The AWS STS [roleArn](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). Example: arn:aws:iam::598203581484:role/test-role | -| `roleSessionName` | String | false | "" (empty string) | The AWS role session name, Name it yourself. | -| `stsEndpoint` | String | false | "" (empty string) | The AWS STS endpoint. By default, the default STS endpoint: https://sts.amazonaws.com is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | false | "" (empty string) | The AWS STS region, By default, the 'region' config or env region is used. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). Example: `arn:aws:events:ap-northeast-1:598263551484:event-bus/my_eventbus` | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -> For details about this connector's advanced features and configurations, see [Advanced features](#advanced-features). - -## Advanced features - -### Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -{% callout title="Note" type="note" %} -Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the support of the Sink downstream system. -{% /callout %} - -### Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -### Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -### Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. - -### Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -### Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -## More Links -- [GitHub Repo](https://github.com/streamnative/pulsar-io-aws-eventbridge) -- [Announcing the Amazon EventBridge Sink Connector for Apache Pulsar](https://streamnative.io/blog/announcing-the-amazon-eventbridge-sink-connector-for-apache-pulsar) -- [Amazon EventBridge connector is now integrated with StreamNative Cloud](https://streamnative.io/blog/amazon-eventbridge-connector-is-now-integrated-with-streamnative-cloud) - - diff --git a/connectors/aws-eventbridge-sink/v2.10.5.7/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v2.10.5.7/aws-eventbridge-sink.md deleted file mode 100644 index 43e9c26e1..000000000 --- a/connectors/aws-eventbridge-sink/v2.10.5.7/aws-eventbridge-sink.md +++ /dev/null @@ -1,346 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: "/images/connectors/aws-eventbridge.png" -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: "true" -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v2.10.5.7/docs/aws-eventbridge-sink.png) - -## Quick start - -### Prerequisites - -The prerequisites for connecting an AWS EventBridge sink connector to external systems include: - -1. Create EventBridge and EventBus in AWS. -2. Create the [AWS User](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) and create `AccessKey`(Please record `AccessKey` and `SecretAccessKey`). -3. Assign permissions to AWS User, and ensure they have the following permissions to the AWS EventBus. For details, see [permissions for event buses](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-bus-perms.html) - -```json - { - "Version": "2012-10-17", - "Statement": [ - - ### To grant permissions for an account to use the PutEvents action, include the following, otherwise delete this section: ### - - { - - "Sid": "AllowAccountToPutEvents", - "Effect": "Allow", - "Principal": { - "AWS": "" - }, - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - }, - - ### Include the following section to grant permissions to all members of your AWS Organizations to use the PutEvents action ### - - { - "Sid": "AllowAllAccountsFromOrganizationToPutEvents", - "Effect": "Allow", - "Principal": "*", - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - "Condition": { - "StringEquals": { - "aws:PrincipalOrgID": "o-yourOrgID" - } - } - } - ] -} -``` -- You can set permissions directly for this user. With this method, when you create a connector, you only need to configure `accessKey` and `secretAccessKey`. -- Or you can use [Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html), this [video](https://www.youtube.com/watch?v=dqF4VJCska4) explains how to use STS on AWS. - With this method, when you create a connector, in addition to configuring `accessKey` and `secretAccessKey`, you also need to configure `role` and `roleSessionName`. - -4. Create a [Rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-rule.html) in EventBridge. -- The data structure sent to Event Bridge is described [here](## Metadata mapping), and you can create **event pattern** based on this structure. -- Set the target according to your needs. If you're testing this connector, you can set the target to [Cloud Watch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html). - - -### 1. Create a connector - -Depending on the environment, there are several ways to create an AWS EventBridge sink connector: - -- [Create Connector on StreamNative Cloud](https://docs.streamnative.io/docs/connector-create). -- [Create Connector with Function worker](https://pulsar.apache.org/docs/3.0.x/io-quickstart/). - Using this way requires you to download a **NAR** package to create a built-in or non-built-in connector. You can download the version you need from [here](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases). -- [Create Connector with Function mesh](https://functionmesh.io/docs/connectors/run-connector). - Using this way requires you to set the docker image. You can choose the version you want to launch from [here](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge/tags) - -No matter how you create an AWS EventBridge sink connector, the minimum configuration contains the following parameters. - -```json - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } -``` -> * The configuration structure varies depending on how you create the AWS EventBridge sink connector. -> For example, some are **JSON**, some are **YAML**, and some are **Kubernetes YAML**. You need to adapt the configs to the corresponding format. -> -> * If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. - -### 2. Send messages to the topic - -{% callout title="Note" type="note" %} -If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information. -{% /callout %} - -``` java - PulsarClient client = PulsarClient.builder() - .serviceUrl("{{Your Pulsar URL}}") - .build(); - - Producer producer = client.newProducer(Schema.STRING) - .topic("{{Your topic name}}") - .create(); - - String message = "{\"msg\": \"msg-data\"}"; - MessageId msgID = producer.send(message); - System.out.println("Publish " + message + " and message ID " + msgID); - - producer.flush(); - producer.close(); - client.close(); -``` - -### 3. Show data on AWS EventBridge -The connector will send the following format of JSON event to EventBridge. -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail-type": "{{Your topic name}}", - "source": "{{Your connector name}}", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "msg": "msg-data" - }, - "message_id": "124:191:0" - } -} -``` - -## Configuration Properties - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | yes | "" (empty string) | The AWS EventBridge [access key ID.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `secretAccessKey` | String | yes | "" (empty string) | The AWS EventBridge [secret access key.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `region` | String | yes | "" (empty string) | The region where AWS EventBridge service is located. [All AWS region](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/Region.html) | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `role` | String | false | "" (empty string) | The AWS STS [roleArn](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). Example: arn:aws:iam::598203581484:role/test-role | -| `roleSessionName` | String | false | "" (empty string) | The AWS role session name, Name it yourself. | -| `stsEndpoint` | String | false | "" (empty string) | The AWS STS endpoint. By default, the default STS endpoint: https://sts.amazonaws.com is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | false | "" (empty string) | The AWS STS region, By default, the 'region' config or env region is used. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). Example: `arn:aws:events:ap-northeast-1:598263551484:event-bus/my_eventbus` | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -> For details about this connector's advanced features and configurations, see [Advanced features](#advanced-features). - -## Advanced features - -### Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -{% callout title="Note" type="note" %} -Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the support of the Sink downstream system. -{% /callout %} - -### Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -### Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -### Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. - -### Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -### Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -## More Links -- [GitHub Repo](https://github.com/streamnative/pulsar-io-aws-eventbridge) -- [Announcing the Amazon EventBridge Sink Connector for Apache Pulsar](https://streamnative.io/blog/announcing-the-amazon-eventbridge-sink-connector-for-apache-pulsar) -- [Amazon EventBridge connector is now integrated with StreamNative Cloud](https://streamnative.io/blog/amazon-eventbridge-connector-is-now-integrated-with-streamnative-cloud) - - diff --git a/connectors/aws-eventbridge-sink/v2.11.1.3/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v2.11.1.3/aws-eventbridge-sink.md deleted file mode 100644 index e3cf6caf8..000000000 --- a/connectors/aws-eventbridge-sink/v2.11.1.3/aws-eventbridge-sink.md +++ /dev/null @@ -1,498 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: /images/connectors/aws-eventbridge.png -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: true -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v2.11.1.3/docs/aws-eventbridge-sink.png) - -# Features - -This section describes the features of the AWS EventBridge sink connector. For details about how to configure these -features, see [how to configure](#how-to-configure). - -## Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -> **Note:** -> Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the -> support of the Sink downstream system. - -## Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -## Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -## Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. -> - -## Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -## Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -# How to get - -This section describes how to get the AWS EventBridge sink connector. - -## Work with Function Worker - -You can get the AWS EventBridge sink connector using one of the following methods if you -use [Pulsar Function Worker](https://pulsar.apache.org/docs/en/functions-worker/) to run the connector in a cluster. - -- Download the NAR package - from [the download page](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases/). - -- Build it from the source code. - -To build the AWS EventBridge sink connector from the source code, follow these steps. - -1. Clone the source code to your machine. - - ```bash - git clone https://github.com/streamnative/pulsar-io-aws-eventbridge - ``` - -2. Build the connector in the `pulsar-io-aws-eventbridge` directory. - - ```bash - mvn clean install -DskipTests - ``` - - After the connector is successfully built, a `NAR` package is generated under the target directory. - - ```bash - ls target - pulsar-io-aws-eventbridge-2.11.1.3.nar - ``` - -## Work with Function Mesh - -You can pull the AWS EventBridge sink connector Docker image from -the [Docker Hub](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge) if you -use [Function Mesh](https://functionmesh.io/docs/connectors/run-connector) to run the connector. - -# How to configure - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | No | "" (empty string) | The EventBridge access key ID. | -| `secretAccessKey` | String | no | "" (empty string) | The EventBridge secret access key. | -| `role` | String | no | "" (empty string) | The AWS role to use. | -| `roleSessionName` | String | no | "" (empty string) | The AWS role session name to use. | -| `stsEndpoint` | String | no | "" (empty string) | The STS endpoint to use. By default, the [default STS endpoint](https://sts.amazonaws.com) is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | no | "" (empty string) | The STS region to use, By default, the 'region' config or env region is used. | -| `region` | String | yes | "" (empty string) | The EventBridge region. | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -## Work with Function Worker - -You can create a configuration file (JSON or YAML) to set the properties if you -use [Pulsar Function Worker](https://pulsar.apache.org/docs/en/functions-worker/) to run connectors in a cluster. - -**Example** - -* JSON - - ```json - { - "name": "eventbridge-sink", - "archive": "connectors/pulsar-io-aws-eventbridge-2.11.1.3.nar", - "tenant": "public", - "namespace": "default", - "inputs": [ - "test-aws-eventbridge-pulsar" - ], - "parallelism": 1, - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } - } - ``` - -* YAML - - ```yaml - name: eventbridge-sink - archive: 'connectors/pulsar-io-aws-eventbridge-2.11.1.3.nar' - tenant: public - namespace: default - inputs: - - test-aws-eventbridge-pulsar - parallelism: 1 - configs: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - ``` - -## Work with Function Mesh - -You can create -a [CustomResourceDefinitions (CRD)](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) -to create an AWS EventBridge sink connector. Using CRD makes Function Mesh naturally integrate with the Kubernetes -ecosystem. For more information about Pulsar sink CRD configurations, -see [sink CRD configurations](https://functionmesh.io/docs/connectors/io-crd-config/sink-crd-config). - -You can define a CRD file (YAML) to set the properties as below. - -```yaml -apiVersion: compute.functionmesh.io/v1alpha1 -kind: Sink -metadata: - name: aws-eventbridge-sink-sample -spec: - image: streamnative/pulsar-io-aws-eventbridge:2.11.1.3 - replicas: 1 - maxReplicas: 1 - input: - topics: - - persistent://public/default/test-aws-eventbridge-pulsar - sinkConfig: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - pulsar: - pulsarConfig: "test-pulsar-sink-config" - resources: - limits: - cpu: "0.2" - memory: 1.1G - requests: - cpu: "0.1" - memory: 1G - java: - jar: connectors/pulsar-io-aws-eventbridge-2.11.1.3.jar - clusterName: test-pulsar - autoAck: false -``` - -# How to use - -You can use the AWS EventBridge sink connector with Function Worker or Function Mesh. - -## Work with Function Worker - -> **Note** -> -> Currently, the AWS EventBridge sink connector cannot run as a built-in connector as it uses the JAR package. - -1. Start a Pulsar cluster in standalone mode. - ``` - PULSAR_HOME/bin/pulsar standalone - ``` - -2. Run the AWS EventBridge sink connector. - ``` - PULSAR_HOME/bin/pulsar-admin sinks localrun \ - --sink-config-file - --archive - ``` - - Or, you can create a connector for the Pulsar cluster. - ``` - PULSAR_HOME/bin/pulsar-admin sinks create \ - --sink-config-file - --archive - ``` - -3. Send messages to a Pulsar topic. - - This example sends ten “hello” messages to the `test-aws-eventbridge-pulsar` topic in the `default` namespace of - the `public` tenant. - - ``` - PULSAR_HOME/bin/pulsar-client produce public/default/test-aws-eventbridge-pulsar --messages hello -n 10 - ``` - -4. Show data on AWS EventBridge. - -The connector will send the following format of JSON event to EventBridge. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": "hello" - } -} -``` - -You can configure the [rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-rules.html) to match events on -Amazon EventBridge, and set the target -to [Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html) to track -data. - -## Work with Function Mesh - -This example describes how to create an AWS EventBridge sink connector for a Kubernetes cluster using Function -Mesh. - -### Prerequisites - -- Create and connect to a [Kubernetes cluster](https://kubernetes.io/). - -- Create a [Pulsar cluster](https://pulsar.apache.org/docs/getting-started-helm/) in the Kubernetes cluster. - -- [Install the Function Mesh Operator and CRD](https://functionmesh.io/docs/install-function-mesh/) into the Kubernetes - cluster. - -- Prepare Amazon EventBridge service. For more details, - see [Getting Started with Amazon EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-setup.html). - -### Steps - -1. Define the AWS EventBridge sink connector with a YAML file and save it as `sink-sample.yaml`. - - This example shows how to publish the AWS EventBridge sink connector to Function Mesh with a Docker image. - ```yaml - apiVersion: compute.functionmesh.io/v1alpha1 - kind: Sink - metadata: - name: aws-eventbridge-sink-sample - spec: - image: streamnative/pulsar-io-aws-eventbridge:2.11.1.3 - replicas: 1 - maxReplicas: 1 - input: - topics: - - persistent://public/default/test-aws-eventbridge-pulsar - sinkConfig: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - pulsar: - pulsarConfig: "test-pulsar-sink-config" - resources: - limits: - cpu: "0.2" - memory: 1.1G - requests: - cpu: "0.1" - memory: 1G - java: - jar: connectors/pulsar-io-aws-eventbridge-2.11.1.3.jar - clusterName: test-pulsar - autoAck: false - ``` - -2. Apply the YAML file to create the AWS EventBridge sink connector. - - **Input** - - ``` - kubectl apply -f - ``` - - **Output** - - ``` - sink.compute.functionmesh.io/aws-eventbridge-sink-sample created - ``` - -3. Check whether the AWS EventBridge sink connector is created successfully. - - **Input** - - ``` - kubectl get all - ``` - - **Output** - - ``` - NAME READY STATUS RESTARTS AGE - pod/aws-eventbridge-sink-sample-0 1/1 Running 0 77s - ``` - - After that, you can produce and consume messages using the AWS EventBridge sink connector between Pulsar and - EventBridge. - - - diff --git a/connectors/aws-eventbridge-sink/v2.11.1.4/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v2.11.1.4/aws-eventbridge-sink.md deleted file mode 100644 index ca9400ded..000000000 --- a/connectors/aws-eventbridge-sink/v2.11.1.4/aws-eventbridge-sink.md +++ /dev/null @@ -1,542 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: /images/connectors/aws-eventbridge.png -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: true -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v2.11.1.4/docs/aws-eventbridge-sink.png) - -# Features - -This section describes the features of the AWS EventBridge sink connector. For details about how to configure these -features, see [how to configure](#how-to-configure). - -## Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -> **Note:** -> Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the -> support of the Sink downstream system. - -## Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -## Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -## Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. -> - -## Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -## Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -# How to get - -This section describes how to get the AWS EventBridge sink connector. - -## Prerequisites - -Prerequisites for an AWS EventBridge sink connector connecting to external systems: - -- Create [Access Keys](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) or [Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) -- Create EventBridge and EventBus in AWS. -- Create a rule in EventBridge. -- Ensure that the account has the following permissions to the AWS EventBus. For details, see [permissions for event buses](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-bus-perms.html) -```json - { - "Version": "2012-10-17", - "Statement": [ - - ### To grant permissions for an account to use the PutEvents action, include the following, otherwise delete this section: ### - - { - - "Sid": "AllowAccountToPutEvents", - "Effect": "Allow", - "Principal": { - "AWS": "" - }, - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - }, - - ### Include the following section to grant permissions to all members of your AWS Organizations to use the PutEvents action ### - - { - "Sid": "AllowAllAccountsFromOrganizationToPutEvents", - "Effect": "Allow", - "Principal": "*", - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - "Condition": { - "StringEquals": { - "aws:PrincipalOrgID": "o-yourOrgID" - } - } - } - ] -} -``` - -## Work with Function Worker - -You can get the AWS EventBridge sink connector using one of the following methods if you -use [Pulsar Function Worker](https://pulsar.apache.org/docs/en/functions-worker/) to run the connector in a cluster. - -- Download the NAR package - from [the download page](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases/). - -- Build it from the source code. - -To build the AWS EventBridge sink connector from the source code, follow these steps. - -1. Clone the source code to your machine. - - ```bash - git clone https://github.com/streamnative/pulsar-io-aws-eventbridge - ``` - -2. Build the connector in the `pulsar-io-aws-eventbridge` directory. - - ```bash - mvn clean install -DskipTests - ``` - - After the connector is successfully built, a `NAR` package is generated under the target directory. - - ```bash - ls target - pulsar-io-aws-eventbridge-2.11.1.4.nar - ``` - -## Work with Function Mesh - -You can pull the AWS EventBridge sink connector Docker image from -the [Docker Hub](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge) if you -use [Function Mesh](https://functionmesh.io/docs/connectors/run-connector) to run the connector. - -# How to configure - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | No | "" (empty string) | The EventBridge access key ID. | -| `secretAccessKey` | String | no | "" (empty string) | The EventBridge secret access key. | -| `role` | String | no | "" (empty string) | The AWS role to use. | -| `roleSessionName` | String | no | "" (empty string) | The AWS role session name to use. | -| `stsEndpoint` | String | no | "" (empty string) | The STS endpoint to use. By default, the [default STS endpoint](https://sts.amazonaws.com) is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | no | "" (empty string) | The STS region to use, By default, the 'region' config or env region is used. | -| `region` | String | yes | "" (empty string) | The EventBridge region. | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -## Work with Function Worker - -You can create a configuration file (JSON or YAML) to set the properties if you -use [Pulsar Function Worker](https://pulsar.apache.org/docs/en/functions-worker/) to run connectors in a cluster. - -**Example** - -* JSON - - ```json - { - "name": "eventbridge-sink", - "archive": "connectors/pulsar-io-aws-eventbridge-2.11.1.4.nar", - "tenant": "public", - "namespace": "default", - "inputs": [ - "test-aws-eventbridge-pulsar" - ], - "parallelism": 1, - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } - } - ``` - -* YAML - - ```yaml - name: eventbridge-sink - archive: 'connectors/pulsar-io-aws-eventbridge-2.11.1.4.nar' - tenant: public - namespace: default - inputs: - - test-aws-eventbridge-pulsar - parallelism: 1 - configs: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - ``` - -## Work with Function Mesh - -You can create -a [CustomResourceDefinitions (CRD)](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) -to create an AWS EventBridge sink connector. Using CRD makes Function Mesh naturally integrate with the Kubernetes -ecosystem. For more information about Pulsar sink CRD configurations, -see [sink CRD configurations](https://functionmesh.io/docs/connectors/io-crd-config/sink-crd-config). - -You can define a CRD file (YAML) to set the properties as below. - -```yaml -apiVersion: compute.functionmesh.io/v1alpha1 -kind: Sink -metadata: - name: aws-eventbridge-sink-sample -spec: - image: streamnative/pulsar-io-aws-eventbridge:2.11.1.4 - replicas: 1 - maxReplicas: 1 - input: - topics: - - persistent://public/default/test-aws-eventbridge-pulsar - sinkConfig: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - pulsar: - pulsarConfig: "test-pulsar-sink-config" - resources: - limits: - cpu: "0.2" - memory: 1.1G - requests: - cpu: "0.1" - memory: 1G - java: - jar: connectors/pulsar-io-aws-eventbridge-2.11.1.4.jar - clusterName: test-pulsar - autoAck: false -``` - -# How to use - -You can use the AWS EventBridge sink connector with Function Worker or Function Mesh. - -## Work with Function Worker - -> **Note** -> -> Currently, the AWS EventBridge sink connector cannot run as a built-in connector as it uses the JAR package. - -1. Start a Pulsar cluster in standalone mode. - ``` - PULSAR_HOME/bin/pulsar standalone - ``` - -2. Run the AWS EventBridge sink connector. - ``` - PULSAR_HOME/bin/pulsar-admin sinks localrun \ - --sink-config-file - --archive - ``` - - Or, you can create a connector for the Pulsar cluster. - ``` - PULSAR_HOME/bin/pulsar-admin sinks create \ - --sink-config-file - --archive - ``` - -3. Send messages to a Pulsar topic. - - This example sends ten “hello” messages to the `test-aws-eventbridge-pulsar` topic in the `default` namespace of - the `public` tenant. - - ``` - PULSAR_HOME/bin/pulsar-client produce public/default/test-aws-eventbridge-pulsar --messages hello -n 10 - ``` - -4. Show data on AWS EventBridge. - -The connector will send the following format of JSON event to EventBridge. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": "hello" - } -} -``` - -You can configure the [rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-rules.html) to match events on -Amazon EventBridge, and set the target -to [Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html) to track -data. - -## Work with Function Mesh - -This example describes how to create an AWS EventBridge sink connector for a Kubernetes cluster using Function -Mesh. - -### Prerequisites - -- Create and connect to a [Kubernetes cluster](https://kubernetes.io/). - -- Create a [Pulsar cluster](https://pulsar.apache.org/docs/getting-started-helm/) in the Kubernetes cluster. - -- [Install the Function Mesh Operator and CRD](https://functionmesh.io/docs/install-function-mesh/) into the Kubernetes - cluster. - -- Prepare Amazon EventBridge service. For more details, - see [Getting Started with Amazon EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-setup.html). - -### Steps - -1. Define the AWS EventBridge sink connector with a YAML file and save it as `sink-sample.yaml`. - - This example shows how to publish the AWS EventBridge sink connector to Function Mesh with a Docker image. - ```yaml - apiVersion: compute.functionmesh.io/v1alpha1 - kind: Sink - metadata: - name: aws-eventbridge-sink-sample - spec: - image: streamnative/pulsar-io-aws-eventbridge:2.11.1.4 - replicas: 1 - maxReplicas: 1 - input: - topics: - - persistent://public/default/test-aws-eventbridge-pulsar - sinkConfig: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - pulsar: - pulsarConfig: "test-pulsar-sink-config" - resources: - limits: - cpu: "0.2" - memory: 1.1G - requests: - cpu: "0.1" - memory: 1G - java: - jar: connectors/pulsar-io-aws-eventbridge-2.11.1.4.jar - clusterName: test-pulsar - autoAck: false - ``` - -2. Apply the YAML file to create the AWS EventBridge sink connector. - - **Input** - - ``` - kubectl apply -f - ``` - - **Output** - - ``` - sink.compute.functionmesh.io/aws-eventbridge-sink-sample created - ``` - -3. Check whether the AWS EventBridge sink connector is created successfully. - - **Input** - - ``` - kubectl get all - ``` - - **Output** - - ``` - NAME READY STATUS RESTARTS AGE - pod/aws-eventbridge-sink-sample-0 1/1 Running 0 77s - ``` - - After that, you can produce and consume messages using the AWS EventBridge sink connector between Pulsar and - EventBridge. - - - diff --git a/connectors/aws-eventbridge-sink/v2.11.2.1/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v2.11.2.1/aws-eventbridge-sink.md deleted file mode 100644 index 1356f9988..000000000 --- a/connectors/aws-eventbridge-sink/v2.11.2.1/aws-eventbridge-sink.md +++ /dev/null @@ -1,542 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: /images/connectors/aws-eventbridge.png -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: true -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v2.11.2.1/docs/aws-eventbridge-sink.png) - -# Features - -This section describes the features of the AWS EventBridge sink connector. For details about how to configure these -features, see [how to configure](#how-to-configure). - -## Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -> **Note:** -> Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the -> support of the Sink downstream system. - -## Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -## Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -## Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. -> - -## Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -## Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -# How to get - -This section describes how to get the AWS EventBridge sink connector. - -## Prerequisites - -Prerequisites for an AWS EventBridge sink connector connecting to external systems: - -- Create [Access Keys](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) or [Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) -- Create EventBridge and EventBus in AWS. -- Create a rule in EventBridge. -- Ensure that the account has the following permissions to the AWS EventBus. For details, see [permissions for event buses](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-bus-perms.html) -```json - { - "Version": "2012-10-17", - "Statement": [ - - ### To grant permissions for an account to use the PutEvents action, include the following, otherwise delete this section: ### - - { - - "Sid": "AllowAccountToPutEvents", - "Effect": "Allow", - "Principal": { - "AWS": "" - }, - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - }, - - ### Include the following section to grant permissions to all members of your AWS Organizations to use the PutEvents action ### - - { - "Sid": "AllowAllAccountsFromOrganizationToPutEvents", - "Effect": "Allow", - "Principal": "*", - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - "Condition": { - "StringEquals": { - "aws:PrincipalOrgID": "o-yourOrgID" - } - } - } - ] -} -``` - -## Work with Function Worker - -You can get the AWS EventBridge sink connector using one of the following methods if you -use [Pulsar Function Worker](https://pulsar.apache.org/docs/en/functions-worker/) to run the connector in a cluster. - -- Download the NAR package - from [the download page](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases/). - -- Build it from the source code. - -To build the AWS EventBridge sink connector from the source code, follow these steps. - -1. Clone the source code to your machine. - - ```bash - git clone https://github.com/streamnative/pulsar-io-aws-eventbridge - ``` - -2. Build the connector in the `pulsar-io-aws-eventbridge` directory. - - ```bash - mvn clean install -DskipTests - ``` - - After the connector is successfully built, a `NAR` package is generated under the target directory. - - ```bash - ls target - pulsar-io-aws-eventbridge-2.11.2.1.nar - ``` - -## Work with Function Mesh - -You can pull the AWS EventBridge sink connector Docker image from -the [Docker Hub](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge) if you -use [Function Mesh](https://functionmesh.io/docs/connectors/run-connector) to run the connector. - -# How to configure - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | No | "" (empty string) | The EventBridge access key ID. | -| `secretAccessKey` | String | no | "" (empty string) | The EventBridge secret access key. | -| `role` | String | no | "" (empty string) | The AWS role to use. | -| `roleSessionName` | String | no | "" (empty string) | The AWS role session name to use. | -| `stsEndpoint` | String | no | "" (empty string) | The STS endpoint to use. By default, the [default STS endpoint](https://sts.amazonaws.com) is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | no | "" (empty string) | The STS region to use, By default, the 'region' config or env region is used. | -| `region` | String | yes | "" (empty string) | The EventBridge region. | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -## Work with Function Worker - -You can create a configuration file (JSON or YAML) to set the properties if you -use [Pulsar Function Worker](https://pulsar.apache.org/docs/en/functions-worker/) to run connectors in a cluster. - -**Example** - -* JSON - - ```json - { - "name": "eventbridge-sink", - "archive": "connectors/pulsar-io-aws-eventbridge-2.11.2.1.nar", - "tenant": "public", - "namespace": "default", - "inputs": [ - "test-aws-eventbridge-pulsar" - ], - "parallelism": 1, - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } - } - ``` - -* YAML - - ```yaml - name: eventbridge-sink - archive: 'connectors/pulsar-io-aws-eventbridge-2.11.2.1.nar' - tenant: public - namespace: default - inputs: - - test-aws-eventbridge-pulsar - parallelism: 1 - configs: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - ``` - -## Work with Function Mesh - -You can create -a [CustomResourceDefinitions (CRD)](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) -to create an AWS EventBridge sink connector. Using CRD makes Function Mesh naturally integrate with the Kubernetes -ecosystem. For more information about Pulsar sink CRD configurations, -see [sink CRD configurations](https://functionmesh.io/docs/connectors/io-crd-config/sink-crd-config). - -You can define a CRD file (YAML) to set the properties as below. - -```yaml -apiVersion: compute.functionmesh.io/v1alpha1 -kind: Sink -metadata: - name: aws-eventbridge-sink-sample -spec: - image: streamnative/pulsar-io-aws-eventbridge:2.11.2.1 - replicas: 1 - maxReplicas: 1 - input: - topics: - - persistent://public/default/test-aws-eventbridge-pulsar - sinkConfig: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - pulsar: - pulsarConfig: "test-pulsar-sink-config" - resources: - limits: - cpu: "0.2" - memory: 1.1G - requests: - cpu: "0.1" - memory: 1G - java: - jar: connectors/pulsar-io-aws-eventbridge-2.11.2.1.jar - clusterName: test-pulsar - autoAck: false -``` - -# How to use - -You can use the AWS EventBridge sink connector with Function Worker or Function Mesh. - -## Work with Function Worker - -> **Note** -> -> Currently, the AWS EventBridge sink connector cannot run as a built-in connector as it uses the JAR package. - -1. Start a Pulsar cluster in standalone mode. - ``` - PULSAR_HOME/bin/pulsar standalone - ``` - -2. Run the AWS EventBridge sink connector. - ``` - PULSAR_HOME/bin/pulsar-admin sinks localrun \ - --sink-config-file - --archive - ``` - - Or, you can create a connector for the Pulsar cluster. - ``` - PULSAR_HOME/bin/pulsar-admin sinks create \ - --sink-config-file - --archive - ``` - -3. Send messages to a Pulsar topic. - - This example sends ten “hello” messages to the `test-aws-eventbridge-pulsar` topic in the `default` namespace of - the `public` tenant. - - ``` - PULSAR_HOME/bin/pulsar-client produce public/default/test-aws-eventbridge-pulsar --messages hello -n 10 - ``` - -4. Show data on AWS EventBridge. - -The connector will send the following format of JSON event to EventBridge. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": "hello" - } -} -``` - -You can configure the [rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-rules.html) to match events on -Amazon EventBridge, and set the target -to [Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html) to track -data. - -## Work with Function Mesh - -This example describes how to create an AWS EventBridge sink connector for a Kubernetes cluster using Function -Mesh. - -### Prerequisites - -- Create and connect to a [Kubernetes cluster](https://kubernetes.io/). - -- Create a [Pulsar cluster](https://pulsar.apache.org/docs/getting-started-helm/) in the Kubernetes cluster. - -- [Install the Function Mesh Operator and CRD](https://functionmesh.io/docs/install-function-mesh/) into the Kubernetes - cluster. - -- Prepare Amazon EventBridge service. For more details, - see [Getting Started with Amazon EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-setup.html). - -### Steps - -1. Define the AWS EventBridge sink connector with a YAML file and save it as `sink-sample.yaml`. - - This example shows how to publish the AWS EventBridge sink connector to Function Mesh with a Docker image. - ```yaml - apiVersion: compute.functionmesh.io/v1alpha1 - kind: Sink - metadata: - name: aws-eventbridge-sink-sample - spec: - image: streamnative/pulsar-io-aws-eventbridge:2.11.2.1 - replicas: 1 - maxReplicas: 1 - input: - topics: - - persistent://public/default/test-aws-eventbridge-pulsar - sinkConfig: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - pulsar: - pulsarConfig: "test-pulsar-sink-config" - resources: - limits: - cpu: "0.2" - memory: 1.1G - requests: - cpu: "0.1" - memory: 1G - java: - jar: connectors/pulsar-io-aws-eventbridge-2.11.2.1.jar - clusterName: test-pulsar - autoAck: false - ``` - -2. Apply the YAML file to create the AWS EventBridge sink connector. - - **Input** - - ``` - kubectl apply -f - ``` - - **Output** - - ``` - sink.compute.functionmesh.io/aws-eventbridge-sink-sample created - ``` - -3. Check whether the AWS EventBridge sink connector is created successfully. - - **Input** - - ``` - kubectl get all - ``` - - **Output** - - ``` - NAME READY STATUS RESTARTS AGE - pod/aws-eventbridge-sink-sample-0 1/1 Running 0 77s - ``` - - After that, you can produce and consume messages using the AWS EventBridge sink connector between Pulsar and - EventBridge. - - - diff --git a/connectors/aws-eventbridge-sink/v2.11.2.10/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v2.11.2.10/aws-eventbridge-sink.md deleted file mode 100644 index 2741797c5..000000000 --- a/connectors/aws-eventbridge-sink/v2.11.2.10/aws-eventbridge-sink.md +++ /dev/null @@ -1,346 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: "/images/connectors/aws-eventbridge.png" -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: "true" -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v2.11.2.10/docs/aws-eventbridge-sink.png) - -## Quick start - -### Prerequisites - -The prerequisites for connecting an AWS EventBridge sink connector to external systems include: - -1. Create EventBridge and EventBus in AWS. -2. Create the [AWS User](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) and create `AccessKey`(Please record `AccessKey` and `SecretAccessKey`). -3. Assign permissions to AWS User, and ensure they have the following permissions to the AWS EventBus. For details, see [permissions for event buses](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-bus-perms.html) - -```json - { - "Version": "2012-10-17", - "Statement": [ - - ### To grant permissions for an account to use the PutEvents action, include the following, otherwise delete this section: ### - - { - - "Sid": "AllowAccountToPutEvents", - "Effect": "Allow", - "Principal": { - "AWS": "" - }, - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - }, - - ### Include the following section to grant permissions to all members of your AWS Organizations to use the PutEvents action ### - - { - "Sid": "AllowAllAccountsFromOrganizationToPutEvents", - "Effect": "Allow", - "Principal": "*", - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - "Condition": { - "StringEquals": { - "aws:PrincipalOrgID": "o-yourOrgID" - } - } - } - ] -} -``` -- You can set permissions directly for this user. With this method, when you create a connector, you only need to configure `accessKey` and `secretAccessKey`. -- Or you can use [Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html), this [video](https://www.youtube.com/watch?v=dqF4VJCska4) explains how to use STS on AWS. - With this method, when you create a connector, in addition to configuring `accessKey` and `secretAccessKey`, you also need to configure `role` and `roleSessionName`. - -4. Create a [Rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-rule.html) in EventBridge. -- The data structure sent to Event Bridge is described [here](## Metadata mapping), and you can create **event pattern** based on this structure. -- Set the target according to your needs. If you're testing this connector, you can set the target to [Cloud Watch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html). - - -### 1. Create a connector - -Depending on the environment, there are several ways to create an AWS EventBridge sink connector: - -- [Create Connector on StreamNative Cloud](https://docs.streamnative.io/docs/connector-create). -- [Create Connector with Function worker](https://pulsar.apache.org/docs/3.0.x/io-quickstart/). - Using this way requires you to download a **NAR** package to create a built-in or non-built-in connector. You can download the version you need from [here](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases). -- [Create Connector with Function mesh](https://functionmesh.io/docs/connectors/run-connector). - Using this way requires you to set the docker image. You can choose the version you want to launch from [here](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge/tags) - -No matter how you create an AWS EventBridge sink connector, the minimum configuration contains the following parameters. - -```json - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } -``` -> * The configuration structure varies depending on how you create the AWS EventBridge sink connector. -> For example, some are **JSON**, some are **YAML**, and some are **Kubernetes YAML**. You need to adapt the configs to the corresponding format. -> -> * If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. - -### 2. Send messages to the topic - -{% callout title="Note" type="note" %} -If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information. -{% /callout %} - -``` java - PulsarClient client = PulsarClient.builder() - .serviceUrl("{{Your Pulsar URL}}") - .build(); - - Producer producer = client.newProducer(Schema.STRING) - .topic("{{Your topic name}}") - .create(); - - String message = "{\"msg\": \"msg-data\"}"; - MessageId msgID = producer.send(message); - System.out.println("Publish " + message + " and message ID " + msgID); - - producer.flush(); - producer.close(); - client.close(); -``` - -### 3. Show data on AWS EventBridge -The connector will send the following format of JSON event to EventBridge. -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail-type": "{{Your topic name}}", - "source": "{{Your connector name}}", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "msg": "msg-data" - }, - "message_id": "124:191:0" - } -} -``` - -## Configuration Properties - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | yes | "" (empty string) | The AWS EventBridge [access key ID.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `secretAccessKey` | String | yes | "" (empty string) | The AWS EventBridge [secret access key.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `region` | String | yes | "" (empty string) | The region where AWS EventBridge service is located. [All AWS region](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/Region.html) | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `role` | String | false | "" (empty string) | The AWS STS [roleArn](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). Example: arn:aws:iam::598203581484:role/test-role | -| `roleSessionName` | String | false | "" (empty string) | The AWS role session name, Name it yourself. | -| `stsEndpoint` | String | false | "" (empty string) | The AWS STS endpoint. By default, the default STS endpoint: https://sts.amazonaws.com is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | false | "" (empty string) | The AWS STS region, By default, the 'region' config or env region is used. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). Example: `arn:aws:events:ap-northeast-1:598263551484:event-bus/my_eventbus` | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -> For details about this connector's advanced features and configurations, see [Advanced features](#advanced-features). - -## Advanced features - -### Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -{% callout title="Note" type="note" %} -Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the support of the Sink downstream system. -{% /callout %} - -### Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -### Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -### Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. - -### Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -### Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -## More Links -- [GitHub Repo](https://github.com/streamnative/pulsar-io-aws-eventbridge) -- [Announcing the Amazon EventBridge Sink Connector for Apache Pulsar](https://streamnative.io/blog/announcing-the-amazon-eventbridge-sink-connector-for-apache-pulsar) -- [Amazon EventBridge connector is now integrated with StreamNative Cloud](https://streamnative.io/blog/amazon-eventbridge-connector-is-now-integrated-with-streamnative-cloud) - - diff --git a/connectors/aws-eventbridge-sink/v2.11.2.11/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v2.11.2.11/aws-eventbridge-sink.md deleted file mode 100644 index 5fffded95..000000000 --- a/connectors/aws-eventbridge-sink/v2.11.2.11/aws-eventbridge-sink.md +++ /dev/null @@ -1,346 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: "/images/connectors/aws-eventbridge.png" -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: "true" -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v2.11.2.11/docs/aws-eventbridge-sink.png) - -## Quick start - -### Prerequisites - -The prerequisites for connecting an AWS EventBridge sink connector to external systems include: - -1. Create EventBridge and EventBus in AWS. -2. Create the [AWS User](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) and create `AccessKey`(Please record `AccessKey` and `SecretAccessKey`). -3. Assign permissions to AWS User, and ensure they have the following permissions to the AWS EventBus. For details, see [permissions for event buses](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-bus-perms.html) - -```json - { - "Version": "2012-10-17", - "Statement": [ - - ### To grant permissions for an account to use the PutEvents action, include the following, otherwise delete this section: ### - - { - - "Sid": "AllowAccountToPutEvents", - "Effect": "Allow", - "Principal": { - "AWS": "" - }, - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - }, - - ### Include the following section to grant permissions to all members of your AWS Organizations to use the PutEvents action ### - - { - "Sid": "AllowAllAccountsFromOrganizationToPutEvents", - "Effect": "Allow", - "Principal": "*", - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - "Condition": { - "StringEquals": { - "aws:PrincipalOrgID": "o-yourOrgID" - } - } - } - ] -} -``` -- You can set permissions directly for this user. With this method, when you create a connector, you only need to configure `accessKey` and `secretAccessKey`. -- Or you can use [Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html), this [video](https://www.youtube.com/watch?v=dqF4VJCska4) explains how to use STS on AWS. - With this method, when you create a connector, in addition to configuring `accessKey` and `secretAccessKey`, you also need to configure `role` and `roleSessionName`. - -4. Create a [Rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-rule.html) in EventBridge. -- The data structure sent to Event Bridge is described [here](## Metadata mapping), and you can create **event pattern** based on this structure. -- Set the target according to your needs. If you're testing this connector, you can set the target to [Cloud Watch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html). - - -### 1. Create a connector - -Depending on the environment, there are several ways to create an AWS EventBridge sink connector: - -- [Create Connector on StreamNative Cloud](https://docs.streamnative.io/docs/connector-create). -- [Create Connector with Function worker](https://pulsar.apache.org/docs/3.0.x/io-quickstart/). - Using this way requires you to download a **NAR** package to create a built-in or non-built-in connector. You can download the version you need from [here](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases). -- [Create Connector with Function mesh](https://functionmesh.io/docs/connectors/run-connector). - Using this way requires you to set the docker image. You can choose the version you want to launch from [here](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge/tags) - -No matter how you create an AWS EventBridge sink connector, the minimum configuration contains the following parameters. - -```json - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } -``` -> * The configuration structure varies depending on how you create the AWS EventBridge sink connector. -> For example, some are **JSON**, some are **YAML**, and some are **Kubernetes YAML**. You need to adapt the configs to the corresponding format. -> -> * If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. - -### 2. Send messages to the topic - -{% callout title="Note" type="note" %} -If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information. -{% /callout %} - -``` java - PulsarClient client = PulsarClient.builder() - .serviceUrl("{{Your Pulsar URL}}") - .build(); - - Producer producer = client.newProducer(Schema.STRING) - .topic("{{Your topic name}}") - .create(); - - String message = "{\"msg\": \"msg-data\"}"; - MessageId msgID = producer.send(message); - System.out.println("Publish " + message + " and message ID " + msgID); - - producer.flush(); - producer.close(); - client.close(); -``` - -### 3. Show data on AWS EventBridge -The connector will send the following format of JSON event to EventBridge. -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail-type": "{{Your topic name}}", - "source": "{{Your connector name}}", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "msg": "msg-data" - }, - "message_id": "124:191:0" - } -} -``` - -## Configuration Properties - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | yes | "" (empty string) | The AWS EventBridge [access key ID.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `secretAccessKey` | String | yes | "" (empty string) | The AWS EventBridge [secret access key.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `region` | String | yes | "" (empty string) | The region where AWS EventBridge service is located. [All AWS region](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/Region.html) | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `role` | String | false | "" (empty string) | The AWS STS [roleArn](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). Example: arn:aws:iam::598203581484:role/test-role | -| `roleSessionName` | String | false | "" (empty string) | The AWS role session name, Name it yourself. | -| `stsEndpoint` | String | false | "" (empty string) | The AWS STS endpoint. By default, the default STS endpoint: https://sts.amazonaws.com is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | false | "" (empty string) | The AWS STS region, By default, the 'region' config or env region is used. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). Example: `arn:aws:events:ap-northeast-1:598263551484:event-bus/my_eventbus` | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -> For details about this connector's advanced features and configurations, see [Advanced features](#advanced-features). - -## Advanced features - -### Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -{% callout title="Note" type="note" %} -Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the support of the Sink downstream system. -{% /callout %} - -### Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -### Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -### Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. - -### Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -### Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -## More Links -- [GitHub Repo](https://github.com/streamnative/pulsar-io-aws-eventbridge) -- [Announcing the Amazon EventBridge Sink Connector for Apache Pulsar](https://streamnative.io/blog/announcing-the-amazon-eventbridge-sink-connector-for-apache-pulsar) -- [Amazon EventBridge connector is now integrated with StreamNative Cloud](https://streamnative.io/blog/amazon-eventbridge-connector-is-now-integrated-with-streamnative-cloud) - - diff --git a/connectors/aws-eventbridge-sink/v2.11.2.12/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v2.11.2.12/aws-eventbridge-sink.md index 006e62c7b..0df9c3583 100644 --- a/connectors/aws-eventbridge-sink/v2.11.2.12/aws-eventbridge-sink.md +++ b/connectors/aws-eventbridge-sink/v2.11.2.12/aws-eventbridge-sink.md @@ -11,13 +11,13 @@ tags: alias: AWS EventBridge Sink Connector features: [""] icon: "/images/connectors/aws-eventbridge.png" -download: +download: Business License support: streamnative -support_link: +support_link: support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" owner_name: "streamnative" owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: +dockerfile: sn_available: "true" id: "aws-eventbridge-sink" --- diff --git a/connectors/aws-eventbridge-sink/v2.11.2.2/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v2.11.2.2/aws-eventbridge-sink.md deleted file mode 100644 index 7eed05788..000000000 --- a/connectors/aws-eventbridge-sink/v2.11.2.2/aws-eventbridge-sink.md +++ /dev/null @@ -1,542 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: /images/connectors/aws-eventbridge.png -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: true -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v2.11.2.2/docs/aws-eventbridge-sink.png) - -# Features - -This section describes the features of the AWS EventBridge sink connector. For details about how to configure these -features, see [how to configure](#how-to-configure). - -## Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -> **Note:** -> Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the -> support of the Sink downstream system. - -## Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -## Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -## Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. -> - -## Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -## Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -# How to get - -This section describes how to get the AWS EventBridge sink connector. - -## Prerequisites - -Prerequisites for an AWS EventBridge sink connector connecting to external systems: - -- Create [Access Keys](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) or [Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) -- Create EventBridge and EventBus in AWS. -- Create a rule in EventBridge. -- Ensure that the account has the following permissions to the AWS EventBus. For details, see [permissions for event buses](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-bus-perms.html) -```json - { - "Version": "2012-10-17", - "Statement": [ - - ### To grant permissions for an account to use the PutEvents action, include the following, otherwise delete this section: ### - - { - - "Sid": "AllowAccountToPutEvents", - "Effect": "Allow", - "Principal": { - "AWS": "" - }, - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - }, - - ### Include the following section to grant permissions to all members of your AWS Organizations to use the PutEvents action ### - - { - "Sid": "AllowAllAccountsFromOrganizationToPutEvents", - "Effect": "Allow", - "Principal": "*", - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - "Condition": { - "StringEquals": { - "aws:PrincipalOrgID": "o-yourOrgID" - } - } - } - ] -} -``` - -## Work with Function Worker - -You can get the AWS EventBridge sink connector using one of the following methods if you -use [Pulsar Function Worker](https://pulsar.apache.org/docs/en/functions-worker/) to run the connector in a cluster. - -- Download the NAR package - from [the download page](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases/). - -- Build it from the source code. - -To build the AWS EventBridge sink connector from the source code, follow these steps. - -1. Clone the source code to your machine. - - ```bash - git clone https://github.com/streamnative/pulsar-io-aws-eventbridge - ``` - -2. Build the connector in the `pulsar-io-aws-eventbridge` directory. - - ```bash - mvn clean install -DskipTests - ``` - - After the connector is successfully built, a `NAR` package is generated under the target directory. - - ```bash - ls target - pulsar-io-aws-eventbridge-2.11.2.2.nar - ``` - -## Work with Function Mesh - -You can pull the AWS EventBridge sink connector Docker image from -the [Docker Hub](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge) if you -use [Function Mesh](https://functionmesh.io/docs/connectors/run-connector) to run the connector. - -# How to configure - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | No | "" (empty string) | The EventBridge access key ID. | -| `secretAccessKey` | String | no | "" (empty string) | The EventBridge secret access key. | -| `role` | String | no | "" (empty string) | The AWS role to use. | -| `roleSessionName` | String | no | "" (empty string) | The AWS role session name to use. | -| `stsEndpoint` | String | no | "" (empty string) | The STS endpoint to use. By default, the [default STS endpoint](https://sts.amazonaws.com) is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | no | "" (empty string) | The STS region to use, By default, the 'region' config or env region is used. | -| `region` | String | yes | "" (empty string) | The EventBridge region. | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -## Work with Function Worker - -You can create a configuration file (JSON or YAML) to set the properties if you -use [Pulsar Function Worker](https://pulsar.apache.org/docs/en/functions-worker/) to run connectors in a cluster. - -**Example** - -* JSON - - ```json - { - "name": "eventbridge-sink", - "archive": "connectors/pulsar-io-aws-eventbridge-2.11.2.2.nar", - "tenant": "public", - "namespace": "default", - "inputs": [ - "test-aws-eventbridge-pulsar" - ], - "parallelism": 1, - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } - } - ``` - -* YAML - - ```yaml - name: eventbridge-sink - archive: 'connectors/pulsar-io-aws-eventbridge-2.11.2.2.nar' - tenant: public - namespace: default - inputs: - - test-aws-eventbridge-pulsar - parallelism: 1 - configs: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - ``` - -## Work with Function Mesh - -You can create -a [CustomResourceDefinitions (CRD)](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) -to create an AWS EventBridge sink connector. Using CRD makes Function Mesh naturally integrate with the Kubernetes -ecosystem. For more information about Pulsar sink CRD configurations, -see [sink CRD configurations](https://functionmesh.io/docs/connectors/io-crd-config/sink-crd-config). - -You can define a CRD file (YAML) to set the properties as below. - -```yaml -apiVersion: compute.functionmesh.io/v1alpha1 -kind: Sink -metadata: - name: aws-eventbridge-sink-sample -spec: - image: streamnative/pulsar-io-aws-eventbridge:2.11.2.2 - replicas: 1 - maxReplicas: 1 - input: - topics: - - persistent://public/default/test-aws-eventbridge-pulsar - sinkConfig: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - pulsar: - pulsarConfig: "test-pulsar-sink-config" - resources: - limits: - cpu: "0.2" - memory: 1.1G - requests: - cpu: "0.1" - memory: 1G - java: - jar: connectors/pulsar-io-aws-eventbridge-2.11.2.2.jar - clusterName: test-pulsar - autoAck: false -``` - -# How to use - -You can use the AWS EventBridge sink connector with Function Worker or Function Mesh. - -## Work with Function Worker - -> **Note** -> -> Currently, the AWS EventBridge sink connector cannot run as a built-in connector as it uses the JAR package. - -1. Start a Pulsar cluster in standalone mode. - ``` - PULSAR_HOME/bin/pulsar standalone - ``` - -2. Run the AWS EventBridge sink connector. - ``` - PULSAR_HOME/bin/pulsar-admin sinks localrun \ - --sink-config-file - --archive - ``` - - Or, you can create a connector for the Pulsar cluster. - ``` - PULSAR_HOME/bin/pulsar-admin sinks create \ - --sink-config-file - --archive - ``` - -3. Send messages to a Pulsar topic. - - This example sends ten “hello” messages to the `test-aws-eventbridge-pulsar` topic in the `default` namespace of - the `public` tenant. - - ``` - PULSAR_HOME/bin/pulsar-client produce public/default/test-aws-eventbridge-pulsar --messages hello -n 10 - ``` - -4. Show data on AWS EventBridge. - -The connector will send the following format of JSON event to EventBridge. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": "hello" - } -} -``` - -You can configure the [rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-rules.html) to match events on -Amazon EventBridge, and set the target -to [Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html) to track -data. - -## Work with Function Mesh - -This example describes how to create an AWS EventBridge sink connector for a Kubernetes cluster using Function -Mesh. - -### Prerequisites - -- Create and connect to a [Kubernetes cluster](https://kubernetes.io/). - -- Create a [Pulsar cluster](https://pulsar.apache.org/docs/getting-started-helm/) in the Kubernetes cluster. - -- [Install the Function Mesh Operator and CRD](https://functionmesh.io/docs/install-function-mesh/) into the Kubernetes - cluster. - -- Prepare Amazon EventBridge service. For more details, - see [Getting Started with Amazon EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-setup.html). - -### Steps - -1. Define the AWS EventBridge sink connector with a YAML file and save it as `sink-sample.yaml`. - - This example shows how to publish the AWS EventBridge sink connector to Function Mesh with a Docker image. - ```yaml - apiVersion: compute.functionmesh.io/v1alpha1 - kind: Sink - metadata: - name: aws-eventbridge-sink-sample - spec: - image: streamnative/pulsar-io-aws-eventbridge:2.11.2.2 - replicas: 1 - maxReplicas: 1 - input: - topics: - - persistent://public/default/test-aws-eventbridge-pulsar - sinkConfig: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - pulsar: - pulsarConfig: "test-pulsar-sink-config" - resources: - limits: - cpu: "0.2" - memory: 1.1G - requests: - cpu: "0.1" - memory: 1G - java: - jar: connectors/pulsar-io-aws-eventbridge-2.11.2.2.jar - clusterName: test-pulsar - autoAck: false - ``` - -2. Apply the YAML file to create the AWS EventBridge sink connector. - - **Input** - - ``` - kubectl apply -f - ``` - - **Output** - - ``` - sink.compute.functionmesh.io/aws-eventbridge-sink-sample created - ``` - -3. Check whether the AWS EventBridge sink connector is created successfully. - - **Input** - - ``` - kubectl get all - ``` - - **Output** - - ``` - NAME READY STATUS RESTARTS AGE - pod/aws-eventbridge-sink-sample-0 1/1 Running 0 77s - ``` - - After that, you can produce and consume messages using the AWS EventBridge sink connector between Pulsar and - EventBridge. - - - diff --git a/connectors/aws-eventbridge-sink/v2.11.2.3/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v2.11.2.3/aws-eventbridge-sink.md deleted file mode 100644 index 8ef56ba45..000000000 --- a/connectors/aws-eventbridge-sink/v2.11.2.3/aws-eventbridge-sink.md +++ /dev/null @@ -1,343 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: "/images/connectors/aws-eventbridge.png" -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: "true" -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v2.11.2.3/docs/aws-eventbridge-sink.png) - -## Quick start - -### Prerequisites - -The prerequisites for connecting an AWS EventBridge sink connector to external systems include: - -1. Create EventBridge and EventBus in AWS. -2. Create the [AWS User](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) and create `AccessKey`(Please record `AccessKey` and `AecretAccessKey`). -3. Assign permissions to Pulsar users, and ensure they have the following permissions to the AWS EventBus. For details, see [permissions for event buses](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-bus-perms.html) - -```json - { - "Version": "2012-10-17", - "Statement": [ - - ### To grant permissions for an account to use the PutEvents action, include the following, otherwise delete this section: ### - - { - - "Sid": "AllowAccountToPutEvents", - "Effect": "Allow", - "Principal": { - "AWS": "" - }, - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - }, - - ### Include the following section to grant permissions to all members of your AWS Organizations to use the PutEvents action ### - - { - "Sid": "AllowAllAccountsFromOrganizationToPutEvents", - "Effect": "Allow", - "Principal": "*", - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - "Condition": { - "StringEquals": { - "aws:PrincipalOrgID": "o-yourOrgID" - } - } - } - ] -} -``` -- You can set permissions directly for this user. With this method, when you create a connector, you only need to configure `accessKey` and `secretAccessKey`. -- Or you can use [Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html), this [video](https://www.youtube.com/watch?v=dqF4VJCska4) explains how to use STS on AWS. - With this method, when you create a connector, in addition to configuring `accessKey` and `secretAccessKey`, you also need to configure `role` and `roleSessionName`. - -4. Create a [Rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-rule.html) in EventBridge. -- The data structure sent to Event Bridge is described [here](## Metadata mapping), and you can create **event pattern** based on this structure. -- Set the target according to your needs. If you're testing this connector, you can set the target to [Cloud Watch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html). - - -### 1. Create a connector - -Depending on the environment, there are several ways to create an AWS EventBridge sink connector: - -- [Create Connector on StreamNative Cloud](https://docs.streamnative.io/docs/connector-create). -- [Create Connector with Function worker](https://pulsar.apache.org/docs/3.0.x/io-quickstart/). - Using this way requires you to download a **NAR** package to create a built-in or non-built-in connector. You can download the version you need from [here](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases). -- [Create Connector with Function mesh](https://functionmesh.io/docs/connectors/run-connector). - Using this way requires you to set the docker image. You can choose the version you want to launch from [here](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge/tags) - -No matter how you create an AWS EventBridge sink connector, the minimum configuration contains the following parameters. - -```json - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } -``` -> * The configuration structure varies depending on how you create the AWS EventBridge sink connector. -> For example, some are **JSON**, some are **YAML**, and some are **Kubernetes YAML**. You need to adapt the configs to the corresponding format. -> -> * If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. - -### 2. Send messages to the topic -> **Note:** If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information. - -``` java - PulsarClient client = PulsarClient.builder() - .serviceUrl("{{Your Pulsar URL}}") - .build(); - - Producer producer = client.newProducer(Schema.STRING) - .topic("{{Your topic name}}") - .create(); - - String message = "{\"msg\": \"msg-data\"}"; - MessageId msgID = producer.send(message); - System.out.println("Publish " + message + " and message ID " + msgID); - - producer.flush(); - producer.close(); - client.close(); -``` - -### 3. Show data on AWS EventBridge -The connector will send the following format of JSON event to EventBridge. -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail-type": "{{Your topic name}}", - "source": "{{Your connector name}}", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "msg": "msg-data" - }, - "message_id": "124:191:0" - } -} -``` - -## Configuration Properties - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | yes | "" (empty string) | The AWS EventBridge [access key ID.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `secretAccessKey` | String | yes | "" (empty string) | The AWS EventBridge [secret access key.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `region` | String | yes | "" (empty string) | The region where AWS EventBridge service is located. [All AWS region](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/Region.html) | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `role` | String | false | "" (empty string) | The AWS STS [roleArn](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). Example: arn:aws:iam::598203581484:role/test-role | -| `roleSessionName` | String | false | "" (empty string) | The AWS role session name, Name it yourself. | -| `stsEndpoint` | String | false | "" (empty string) | The AWS STS endpoint. By default, the default STS endpoint: https://sts.amazonaws.com is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | false | "" (empty string) | The AWS STS region, By default, the 'region' config or env region is used. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). Example: `arn:aws:events:ap-northeast-1:598263551484:event-bus/my_eventbus` | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -> For details about this connector's advanced features and configurations, see [Advanced features](#advanced-features). - -## Advanced features - -### Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -> **Note:** -> Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the -> support of the Sink downstream system. - -### Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -### Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -### Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. - -### Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -### Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -## More Links -- [GitHub Repo](https://github.com/streamnative/pulsar-io-aws-eventbridge) -- [Announcing the Amazon EventBridge Sink Connector for Apache Pulsar](https://streamnative.io/blog/announcing-the-amazon-eventbridge-sink-connector-for-apache-pulsar) -- [Amazon EventBridge connector is now integrated with StreamNative Cloud](https://streamnative.io/blog/amazon-eventbridge-connector-is-now-integrated-with-streamnative-cloud) - - diff --git a/connectors/aws-eventbridge-sink/v2.11.2.4/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v2.11.2.4/aws-eventbridge-sink.md deleted file mode 100644 index 10e7f60af..000000000 --- a/connectors/aws-eventbridge-sink/v2.11.2.4/aws-eventbridge-sink.md +++ /dev/null @@ -1,343 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: "/images/connectors/aws-eventbridge.png" -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: "true" -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v2.11.2.4/docs/aws-eventbridge-sink.png) - -## Quick start - -### Prerequisites - -The prerequisites for connecting an AWS EventBridge sink connector to external systems include: - -1. Create EventBridge and EventBus in AWS. -2. Create the [AWS User](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) and create `AccessKey`(Please record `AccessKey` and `AecretAccessKey`). -3. Assign permissions to Pulsar users, and ensure they have the following permissions to the AWS EventBus. For details, see [permissions for event buses](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-bus-perms.html) - -```json - { - "Version": "2012-10-17", - "Statement": [ - - ### To grant permissions for an account to use the PutEvents action, include the following, otherwise delete this section: ### - - { - - "Sid": "AllowAccountToPutEvents", - "Effect": "Allow", - "Principal": { - "AWS": "" - }, - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - }, - - ### Include the following section to grant permissions to all members of your AWS Organizations to use the PutEvents action ### - - { - "Sid": "AllowAllAccountsFromOrganizationToPutEvents", - "Effect": "Allow", - "Principal": "*", - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - "Condition": { - "StringEquals": { - "aws:PrincipalOrgID": "o-yourOrgID" - } - } - } - ] -} -``` -- You can set permissions directly for this user. With this method, when you create a connector, you only need to configure `accessKey` and `secretAccessKey`. -- Or you can use [Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html), this [video](https://www.youtube.com/watch?v=dqF4VJCska4) explains how to use STS on AWS. - With this method, when you create a connector, in addition to configuring `accessKey` and `secretAccessKey`, you also need to configure `role` and `roleSessionName`. - -4. Create a [Rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-rule.html) in EventBridge. -- The data structure sent to Event Bridge is described [here](## Metadata mapping), and you can create **event pattern** based on this structure. -- Set the target according to your needs. If you're testing this connector, you can set the target to [Cloud Watch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html). - - -### 1. Create a connector - -Depending on the environment, there are several ways to create an AWS EventBridge sink connector: - -- [Create Connector on StreamNative Cloud](https://docs.streamnative.io/docs/connector-create). -- [Create Connector with Function worker](https://pulsar.apache.org/docs/3.0.x/io-quickstart/). - Using this way requires you to download a **NAR** package to create a built-in or non-built-in connector. You can download the version you need from [here](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases). -- [Create Connector with Function mesh](https://functionmesh.io/docs/connectors/run-connector). - Using this way requires you to set the docker image. You can choose the version you want to launch from [here](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge/tags) - -No matter how you create an AWS EventBridge sink connector, the minimum configuration contains the following parameters. - -```json - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } -``` -> * The configuration structure varies depending on how you create the AWS EventBridge sink connector. -> For example, some are **JSON**, some are **YAML**, and some are **Kubernetes YAML**. You need to adapt the configs to the corresponding format. -> -> * If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. - -### 2. Send messages to the topic -> **Note:** If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information. - -``` java - PulsarClient client = PulsarClient.builder() - .serviceUrl("{{Your Pulsar URL}}") - .build(); - - Producer producer = client.newProducer(Schema.STRING) - .topic("{{Your topic name}}") - .create(); - - String message = "{\"msg\": \"msg-data\"}"; - MessageId msgID = producer.send(message); - System.out.println("Publish " + message + " and message ID " + msgID); - - producer.flush(); - producer.close(); - client.close(); -``` - -### 3. Show data on AWS EventBridge -The connector will send the following format of JSON event to EventBridge. -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail-type": "{{Your topic name}}", - "source": "{{Your connector name}}", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "msg": "msg-data" - }, - "message_id": "124:191:0" - } -} -``` - -## Configuration Properties - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | yes | "" (empty string) | The AWS EventBridge [access key ID.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `secretAccessKey` | String | yes | "" (empty string) | The AWS EventBridge [secret access key.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `region` | String | yes | "" (empty string) | The region where AWS EventBridge service is located. [All AWS region](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/Region.html) | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `role` | String | false | "" (empty string) | The AWS STS [roleArn](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). Example: arn:aws:iam::598203581484:role/test-role | -| `roleSessionName` | String | false | "" (empty string) | The AWS role session name, Name it yourself. | -| `stsEndpoint` | String | false | "" (empty string) | The AWS STS endpoint. By default, the default STS endpoint: https://sts.amazonaws.com is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | false | "" (empty string) | The AWS STS region, By default, the 'region' config or env region is used. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). Example: `arn:aws:events:ap-northeast-1:598263551484:event-bus/my_eventbus` | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -> For details about this connector's advanced features and configurations, see [Advanced features](#advanced-features). - -## Advanced features - -### Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -> **Note:** -> Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the -> support of the Sink downstream system. - -### Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -### Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -### Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. - -### Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -### Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -## More Links -- [GitHub Repo](https://github.com/streamnative/pulsar-io-aws-eventbridge) -- [Announcing the Amazon EventBridge Sink Connector for Apache Pulsar](https://streamnative.io/blog/announcing-the-amazon-eventbridge-sink-connector-for-apache-pulsar) -- [Amazon EventBridge connector is now integrated with StreamNative Cloud](https://streamnative.io/blog/amazon-eventbridge-connector-is-now-integrated-with-streamnative-cloud) - - diff --git a/connectors/aws-eventbridge-sink/v2.11.2.5/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v2.11.2.5/aws-eventbridge-sink.md deleted file mode 100644 index bfa6d87be..000000000 --- a/connectors/aws-eventbridge-sink/v2.11.2.5/aws-eventbridge-sink.md +++ /dev/null @@ -1,346 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: "/images/connectors/aws-eventbridge.png" -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: "true" -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v2.11.2.5/docs/aws-eventbridge-sink.png) - -## Quick start - -### Prerequisites - -The prerequisites for connecting an AWS EventBridge sink connector to external systems include: - -1. Create EventBridge and EventBus in AWS. -2. Create the [AWS User](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) and create `AccessKey`(Please record `AccessKey` and `SecretAccessKey`). -3. Assign permissions to AWS User, and ensure they have the following permissions to the AWS EventBus. For details, see [permissions for event buses](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-bus-perms.html) - -```json - { - "Version": "2012-10-17", - "Statement": [ - - ### To grant permissions for an account to use the PutEvents action, include the following, otherwise delete this section: ### - - { - - "Sid": "AllowAccountToPutEvents", - "Effect": "Allow", - "Principal": { - "AWS": "" - }, - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - }, - - ### Include the following section to grant permissions to all members of your AWS Organizations to use the PutEvents action ### - - { - "Sid": "AllowAllAccountsFromOrganizationToPutEvents", - "Effect": "Allow", - "Principal": "*", - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - "Condition": { - "StringEquals": { - "aws:PrincipalOrgID": "o-yourOrgID" - } - } - } - ] -} -``` -- You can set permissions directly for this user. With this method, when you create a connector, you only need to configure `accessKey` and `secretAccessKey`. -- Or you can use [Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html), this [video](https://www.youtube.com/watch?v=dqF4VJCska4) explains how to use STS on AWS. - With this method, when you create a connector, in addition to configuring `accessKey` and `secretAccessKey`, you also need to configure `role` and `roleSessionName`. - -4. Create a [Rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-rule.html) in EventBridge. -- The data structure sent to Event Bridge is described [here](## Metadata mapping), and you can create **event pattern** based on this structure. -- Set the target according to your needs. If you're testing this connector, you can set the target to [Cloud Watch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html). - - -### 1. Create a connector - -Depending on the environment, there are several ways to create an AWS EventBridge sink connector: - -- [Create Connector on StreamNative Cloud](https://docs.streamnative.io/docs/connector-create). -- [Create Connector with Function worker](https://pulsar.apache.org/docs/3.0.x/io-quickstart/). - Using this way requires you to download a **NAR** package to create a built-in or non-built-in connector. You can download the version you need from [here](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases). -- [Create Connector with Function mesh](https://functionmesh.io/docs/connectors/run-connector). - Using this way requires you to set the docker image. You can choose the version you want to launch from [here](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge/tags) - -No matter how you create an AWS EventBridge sink connector, the minimum configuration contains the following parameters. - -```json - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } -``` -> * The configuration structure varies depending on how you create the AWS EventBridge sink connector. -> For example, some are **JSON**, some are **YAML**, and some are **Kubernetes YAML**. You need to adapt the configs to the corresponding format. -> -> * If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. - -### 2. Send messages to the topic - -{% callout title="Note" type="note" %} -If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information. -{% /callout %} - -``` java - PulsarClient client = PulsarClient.builder() - .serviceUrl("{{Your Pulsar URL}}") - .build(); - - Producer producer = client.newProducer(Schema.STRING) - .topic("{{Your topic name}}") - .create(); - - String message = "{\"msg\": \"msg-data\"}"; - MessageId msgID = producer.send(message); - System.out.println("Publish " + message + " and message ID " + msgID); - - producer.flush(); - producer.close(); - client.close(); -``` - -### 3. Show data on AWS EventBridge -The connector will send the following format of JSON event to EventBridge. -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail-type": "{{Your topic name}}", - "source": "{{Your connector name}}", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "msg": "msg-data" - }, - "message_id": "124:191:0" - } -} -``` - -## Configuration Properties - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | yes | "" (empty string) | The AWS EventBridge [access key ID.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `secretAccessKey` | String | yes | "" (empty string) | The AWS EventBridge [secret access key.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `region` | String | yes | "" (empty string) | The region where AWS EventBridge service is located. [All AWS region](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/Region.html) | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `role` | String | false | "" (empty string) | The AWS STS [roleArn](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). Example: arn:aws:iam::598203581484:role/test-role | -| `roleSessionName` | String | false | "" (empty string) | The AWS role session name, Name it yourself. | -| `stsEndpoint` | String | false | "" (empty string) | The AWS STS endpoint. By default, the default STS endpoint: https://sts.amazonaws.com is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | false | "" (empty string) | The AWS STS region, By default, the 'region' config or env region is used. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). Example: `arn:aws:events:ap-northeast-1:598263551484:event-bus/my_eventbus` | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -> For details about this connector's advanced features and configurations, see [Advanced features](#advanced-features). - -## Advanced features - -### Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -{% callout title="Note" type="note" %} -Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the support of the Sink downstream system. -{% /callout %} - -### Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -### Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -### Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. - -### Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -### Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -## More Links -- [GitHub Repo](https://github.com/streamnative/pulsar-io-aws-eventbridge) -- [Announcing the Amazon EventBridge Sink Connector for Apache Pulsar](https://streamnative.io/blog/announcing-the-amazon-eventbridge-sink-connector-for-apache-pulsar) -- [Amazon EventBridge connector is now integrated with StreamNative Cloud](https://streamnative.io/blog/amazon-eventbridge-connector-is-now-integrated-with-streamnative-cloud) - - diff --git a/connectors/aws-eventbridge-sink/v2.11.2.6/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v2.11.2.6/aws-eventbridge-sink.md deleted file mode 100644 index 3407bc261..000000000 --- a/connectors/aws-eventbridge-sink/v2.11.2.6/aws-eventbridge-sink.md +++ /dev/null @@ -1,346 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: "/images/connectors/aws-eventbridge.png" -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: "true" -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v2.11.2.6/docs/aws-eventbridge-sink.png) - -## Quick start - -### Prerequisites - -The prerequisites for connecting an AWS EventBridge sink connector to external systems include: - -1. Create EventBridge and EventBus in AWS. -2. Create the [AWS User](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) and create `AccessKey`(Please record `AccessKey` and `SecretAccessKey`). -3. Assign permissions to AWS User, and ensure they have the following permissions to the AWS EventBus. For details, see [permissions for event buses](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-bus-perms.html) - -```json - { - "Version": "2012-10-17", - "Statement": [ - - ### To grant permissions for an account to use the PutEvents action, include the following, otherwise delete this section: ### - - { - - "Sid": "AllowAccountToPutEvents", - "Effect": "Allow", - "Principal": { - "AWS": "" - }, - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - }, - - ### Include the following section to grant permissions to all members of your AWS Organizations to use the PutEvents action ### - - { - "Sid": "AllowAllAccountsFromOrganizationToPutEvents", - "Effect": "Allow", - "Principal": "*", - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - "Condition": { - "StringEquals": { - "aws:PrincipalOrgID": "o-yourOrgID" - } - } - } - ] -} -``` -- You can set permissions directly for this user. With this method, when you create a connector, you only need to configure `accessKey` and `secretAccessKey`. -- Or you can use [Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html), this [video](https://www.youtube.com/watch?v=dqF4VJCska4) explains how to use STS on AWS. - With this method, when you create a connector, in addition to configuring `accessKey` and `secretAccessKey`, you also need to configure `role` and `roleSessionName`. - -4. Create a [Rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-rule.html) in EventBridge. -- The data structure sent to Event Bridge is described [here](## Metadata mapping), and you can create **event pattern** based on this structure. -- Set the target according to your needs. If you're testing this connector, you can set the target to [Cloud Watch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html). - - -### 1. Create a connector - -Depending on the environment, there are several ways to create an AWS EventBridge sink connector: - -- [Create Connector on StreamNative Cloud](https://docs.streamnative.io/docs/connector-create). -- [Create Connector with Function worker](https://pulsar.apache.org/docs/3.0.x/io-quickstart/). - Using this way requires you to download a **NAR** package to create a built-in or non-built-in connector. You can download the version you need from [here](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases). -- [Create Connector with Function mesh](https://functionmesh.io/docs/connectors/run-connector). - Using this way requires you to set the docker image. You can choose the version you want to launch from [here](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge/tags) - -No matter how you create an AWS EventBridge sink connector, the minimum configuration contains the following parameters. - -```json - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } -``` -> * The configuration structure varies depending on how you create the AWS EventBridge sink connector. -> For example, some are **JSON**, some are **YAML**, and some are **Kubernetes YAML**. You need to adapt the configs to the corresponding format. -> -> * If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. - -### 2. Send messages to the topic - -{% callout title="Note" type="note" %} -If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information. -{% /callout %} - -``` java - PulsarClient client = PulsarClient.builder() - .serviceUrl("{{Your Pulsar URL}}") - .build(); - - Producer producer = client.newProducer(Schema.STRING) - .topic("{{Your topic name}}") - .create(); - - String message = "{\"msg\": \"msg-data\"}"; - MessageId msgID = producer.send(message); - System.out.println("Publish " + message + " and message ID " + msgID); - - producer.flush(); - producer.close(); - client.close(); -``` - -### 3. Show data on AWS EventBridge -The connector will send the following format of JSON event to EventBridge. -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail-type": "{{Your topic name}}", - "source": "{{Your connector name}}", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "msg": "msg-data" - }, - "message_id": "124:191:0" - } -} -``` - -## Configuration Properties - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | yes | "" (empty string) | The AWS EventBridge [access key ID.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `secretAccessKey` | String | yes | "" (empty string) | The AWS EventBridge [secret access key.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `region` | String | yes | "" (empty string) | The region where AWS EventBridge service is located. [All AWS region](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/Region.html) | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `role` | String | false | "" (empty string) | The AWS STS [roleArn](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). Example: arn:aws:iam::598203581484:role/test-role | -| `roleSessionName` | String | false | "" (empty string) | The AWS role session name, Name it yourself. | -| `stsEndpoint` | String | false | "" (empty string) | The AWS STS endpoint. By default, the default STS endpoint: https://sts.amazonaws.com is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | false | "" (empty string) | The AWS STS region, By default, the 'region' config or env region is used. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). Example: `arn:aws:events:ap-northeast-1:598263551484:event-bus/my_eventbus` | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -> For details about this connector's advanced features and configurations, see [Advanced features](#advanced-features). - -## Advanced features - -### Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -{% callout title="Note" type="note" %} -Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the support of the Sink downstream system. -{% /callout %} - -### Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -### Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -### Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. - -### Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -### Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -## More Links -- [GitHub Repo](https://github.com/streamnative/pulsar-io-aws-eventbridge) -- [Announcing the Amazon EventBridge Sink Connector for Apache Pulsar](https://streamnative.io/blog/announcing-the-amazon-eventbridge-sink-connector-for-apache-pulsar) -- [Amazon EventBridge connector is now integrated with StreamNative Cloud](https://streamnative.io/blog/amazon-eventbridge-connector-is-now-integrated-with-streamnative-cloud) - - diff --git a/connectors/aws-eventbridge-sink/v2.11.2.7/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v2.11.2.7/aws-eventbridge-sink.md deleted file mode 100644 index b60a3a419..000000000 --- a/connectors/aws-eventbridge-sink/v2.11.2.7/aws-eventbridge-sink.md +++ /dev/null @@ -1,346 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: "/images/connectors/aws-eventbridge.png" -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: "true" -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v2.11.2.7/docs/aws-eventbridge-sink.png) - -## Quick start - -### Prerequisites - -The prerequisites for connecting an AWS EventBridge sink connector to external systems include: - -1. Create EventBridge and EventBus in AWS. -2. Create the [AWS User](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) and create `AccessKey`(Please record `AccessKey` and `SecretAccessKey`). -3. Assign permissions to AWS User, and ensure they have the following permissions to the AWS EventBus. For details, see [permissions for event buses](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-bus-perms.html) - -```json - { - "Version": "2012-10-17", - "Statement": [ - - ### To grant permissions for an account to use the PutEvents action, include the following, otherwise delete this section: ### - - { - - "Sid": "AllowAccountToPutEvents", - "Effect": "Allow", - "Principal": { - "AWS": "" - }, - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - }, - - ### Include the following section to grant permissions to all members of your AWS Organizations to use the PutEvents action ### - - { - "Sid": "AllowAllAccountsFromOrganizationToPutEvents", - "Effect": "Allow", - "Principal": "*", - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - "Condition": { - "StringEquals": { - "aws:PrincipalOrgID": "o-yourOrgID" - } - } - } - ] -} -``` -- You can set permissions directly for this user. With this method, when you create a connector, you only need to configure `accessKey` and `secretAccessKey`. -- Or you can use [Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html), this [video](https://www.youtube.com/watch?v=dqF4VJCska4) explains how to use STS on AWS. - With this method, when you create a connector, in addition to configuring `accessKey` and `secretAccessKey`, you also need to configure `role` and `roleSessionName`. - -4. Create a [Rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-rule.html) in EventBridge. -- The data structure sent to Event Bridge is described [here](## Metadata mapping), and you can create **event pattern** based on this structure. -- Set the target according to your needs. If you're testing this connector, you can set the target to [Cloud Watch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html). - - -### 1. Create a connector - -Depending on the environment, there are several ways to create an AWS EventBridge sink connector: - -- [Create Connector on StreamNative Cloud](https://docs.streamnative.io/docs/connector-create). -- [Create Connector with Function worker](https://pulsar.apache.org/docs/3.0.x/io-quickstart/). - Using this way requires you to download a **NAR** package to create a built-in or non-built-in connector. You can download the version you need from [here](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases). -- [Create Connector with Function mesh](https://functionmesh.io/docs/connectors/run-connector). - Using this way requires you to set the docker image. You can choose the version you want to launch from [here](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge/tags) - -No matter how you create an AWS EventBridge sink connector, the minimum configuration contains the following parameters. - -```json - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } -``` -> * The configuration structure varies depending on how you create the AWS EventBridge sink connector. -> For example, some are **JSON**, some are **YAML**, and some are **Kubernetes YAML**. You need to adapt the configs to the corresponding format. -> -> * If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. - -### 2. Send messages to the topic - -{% callout title="Note" type="note" %} -If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information. -{% /callout %} - -``` java - PulsarClient client = PulsarClient.builder() - .serviceUrl("{{Your Pulsar URL}}") - .build(); - - Producer producer = client.newProducer(Schema.STRING) - .topic("{{Your topic name}}") - .create(); - - String message = "{\"msg\": \"msg-data\"}"; - MessageId msgID = producer.send(message); - System.out.println("Publish " + message + " and message ID " + msgID); - - producer.flush(); - producer.close(); - client.close(); -``` - -### 3. Show data on AWS EventBridge -The connector will send the following format of JSON event to EventBridge. -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail-type": "{{Your topic name}}", - "source": "{{Your connector name}}", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "msg": "msg-data" - }, - "message_id": "124:191:0" - } -} -``` - -## Configuration Properties - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | yes | "" (empty string) | The AWS EventBridge [access key ID.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `secretAccessKey` | String | yes | "" (empty string) | The AWS EventBridge [secret access key.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `region` | String | yes | "" (empty string) | The region where AWS EventBridge service is located. [All AWS region](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/Region.html) | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `role` | String | false | "" (empty string) | The AWS STS [roleArn](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). Example: arn:aws:iam::598203581484:role/test-role | -| `roleSessionName` | String | false | "" (empty string) | The AWS role session name, Name it yourself. | -| `stsEndpoint` | String | false | "" (empty string) | The AWS STS endpoint. By default, the default STS endpoint: https://sts.amazonaws.com is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | false | "" (empty string) | The AWS STS region, By default, the 'region' config or env region is used. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). Example: `arn:aws:events:ap-northeast-1:598263551484:event-bus/my_eventbus` | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -> For details about this connector's advanced features and configurations, see [Advanced features](#advanced-features). - -## Advanced features - -### Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -{% callout title="Note" type="note" %} -Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the support of the Sink downstream system. -{% /callout %} - -### Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -### Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -### Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. - -### Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -### Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -## More Links -- [GitHub Repo](https://github.com/streamnative/pulsar-io-aws-eventbridge) -- [Announcing the Amazon EventBridge Sink Connector for Apache Pulsar](https://streamnative.io/blog/announcing-the-amazon-eventbridge-sink-connector-for-apache-pulsar) -- [Amazon EventBridge connector is now integrated with StreamNative Cloud](https://streamnative.io/blog/amazon-eventbridge-connector-is-now-integrated-with-streamnative-cloud) - - diff --git a/connectors/aws-eventbridge-sink/v2.11.2.8/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v2.11.2.8/aws-eventbridge-sink.md deleted file mode 100644 index da461f7af..000000000 --- a/connectors/aws-eventbridge-sink/v2.11.2.8/aws-eventbridge-sink.md +++ /dev/null @@ -1,346 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: "/images/connectors/aws-eventbridge.png" -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: "true" -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v2.11.2.8/docs/aws-eventbridge-sink.png) - -## Quick start - -### Prerequisites - -The prerequisites for connecting an AWS EventBridge sink connector to external systems include: - -1. Create EventBridge and EventBus in AWS. -2. Create the [AWS User](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) and create `AccessKey`(Please record `AccessKey` and `SecretAccessKey`). -3. Assign permissions to AWS User, and ensure they have the following permissions to the AWS EventBus. For details, see [permissions for event buses](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-bus-perms.html) - -```json - { - "Version": "2012-10-17", - "Statement": [ - - ### To grant permissions for an account to use the PutEvents action, include the following, otherwise delete this section: ### - - { - - "Sid": "AllowAccountToPutEvents", - "Effect": "Allow", - "Principal": { - "AWS": "" - }, - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - }, - - ### Include the following section to grant permissions to all members of your AWS Organizations to use the PutEvents action ### - - { - "Sid": "AllowAllAccountsFromOrganizationToPutEvents", - "Effect": "Allow", - "Principal": "*", - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - "Condition": { - "StringEquals": { - "aws:PrincipalOrgID": "o-yourOrgID" - } - } - } - ] -} -``` -- You can set permissions directly for this user. With this method, when you create a connector, you only need to configure `accessKey` and `secretAccessKey`. -- Or you can use [Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html), this [video](https://www.youtube.com/watch?v=dqF4VJCska4) explains how to use STS on AWS. - With this method, when you create a connector, in addition to configuring `accessKey` and `secretAccessKey`, you also need to configure `role` and `roleSessionName`. - -4. Create a [Rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-rule.html) in EventBridge. -- The data structure sent to Event Bridge is described [here](## Metadata mapping), and you can create **event pattern** based on this structure. -- Set the target according to your needs. If you're testing this connector, you can set the target to [Cloud Watch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html). - - -### 1. Create a connector - -Depending on the environment, there are several ways to create an AWS EventBridge sink connector: - -- [Create Connector on StreamNative Cloud](https://docs.streamnative.io/docs/connector-create). -- [Create Connector with Function worker](https://pulsar.apache.org/docs/3.0.x/io-quickstart/). - Using this way requires you to download a **NAR** package to create a built-in or non-built-in connector. You can download the version you need from [here](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases). -- [Create Connector with Function mesh](https://functionmesh.io/docs/connectors/run-connector). - Using this way requires you to set the docker image. You can choose the version you want to launch from [here](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge/tags) - -No matter how you create an AWS EventBridge sink connector, the minimum configuration contains the following parameters. - -```json - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } -``` -> * The configuration structure varies depending on how you create the AWS EventBridge sink connector. -> For example, some are **JSON**, some are **YAML**, and some are **Kubernetes YAML**. You need to adapt the configs to the corresponding format. -> -> * If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. - -### 2. Send messages to the topic - -{% callout title="Note" type="note" %} -If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information. -{% /callout %} - -``` java - PulsarClient client = PulsarClient.builder() - .serviceUrl("{{Your Pulsar URL}}") - .build(); - - Producer producer = client.newProducer(Schema.STRING) - .topic("{{Your topic name}}") - .create(); - - String message = "{\"msg\": \"msg-data\"}"; - MessageId msgID = producer.send(message); - System.out.println("Publish " + message + " and message ID " + msgID); - - producer.flush(); - producer.close(); - client.close(); -``` - -### 3. Show data on AWS EventBridge -The connector will send the following format of JSON event to EventBridge. -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail-type": "{{Your topic name}}", - "source": "{{Your connector name}}", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "msg": "msg-data" - }, - "message_id": "124:191:0" - } -} -``` - -## Configuration Properties - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | yes | "" (empty string) | The AWS EventBridge [access key ID.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `secretAccessKey` | String | yes | "" (empty string) | The AWS EventBridge [secret access key.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `region` | String | yes | "" (empty string) | The region where AWS EventBridge service is located. [All AWS region](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/Region.html) | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `role` | String | false | "" (empty string) | The AWS STS [roleArn](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). Example: arn:aws:iam::598203581484:role/test-role | -| `roleSessionName` | String | false | "" (empty string) | The AWS role session name, Name it yourself. | -| `stsEndpoint` | String | false | "" (empty string) | The AWS STS endpoint. By default, the default STS endpoint: https://sts.amazonaws.com is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | false | "" (empty string) | The AWS STS region, By default, the 'region' config or env region is used. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). Example: `arn:aws:events:ap-northeast-1:598263551484:event-bus/my_eventbus` | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -> For details about this connector's advanced features and configurations, see [Advanced features](#advanced-features). - -## Advanced features - -### Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -{% callout title="Note" type="note" %} -Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the support of the Sink downstream system. -{% /callout %} - -### Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -### Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -### Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. - -### Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -### Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -## More Links -- [GitHub Repo](https://github.com/streamnative/pulsar-io-aws-eventbridge) -- [Announcing the Amazon EventBridge Sink Connector for Apache Pulsar](https://streamnative.io/blog/announcing-the-amazon-eventbridge-sink-connector-for-apache-pulsar) -- [Amazon EventBridge connector is now integrated with StreamNative Cloud](https://streamnative.io/blog/amazon-eventbridge-connector-is-now-integrated-with-streamnative-cloud) - - diff --git a/connectors/aws-eventbridge-sink/v2.11.2.9/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v2.11.2.9/aws-eventbridge-sink.md deleted file mode 100644 index b41fa966d..000000000 --- a/connectors/aws-eventbridge-sink/v2.11.2.9/aws-eventbridge-sink.md +++ /dev/null @@ -1,346 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: "/images/connectors/aws-eventbridge.png" -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: "true" -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v2.11.2.9/docs/aws-eventbridge-sink.png) - -## Quick start - -### Prerequisites - -The prerequisites for connecting an AWS EventBridge sink connector to external systems include: - -1. Create EventBridge and EventBus in AWS. -2. Create the [AWS User](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) and create `AccessKey`(Please record `AccessKey` and `SecretAccessKey`). -3. Assign permissions to AWS User, and ensure they have the following permissions to the AWS EventBus. For details, see [permissions for event buses](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-bus-perms.html) - -```json - { - "Version": "2012-10-17", - "Statement": [ - - ### To grant permissions for an account to use the PutEvents action, include the following, otherwise delete this section: ### - - { - - "Sid": "AllowAccountToPutEvents", - "Effect": "Allow", - "Principal": { - "AWS": "" - }, - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - }, - - ### Include the following section to grant permissions to all members of your AWS Organizations to use the PutEvents action ### - - { - "Sid": "AllowAllAccountsFromOrganizationToPutEvents", - "Effect": "Allow", - "Principal": "*", - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - "Condition": { - "StringEquals": { - "aws:PrincipalOrgID": "o-yourOrgID" - } - } - } - ] -} -``` -- You can set permissions directly for this user. With this method, when you create a connector, you only need to configure `accessKey` and `secretAccessKey`. -- Or you can use [Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html), this [video](https://www.youtube.com/watch?v=dqF4VJCska4) explains how to use STS on AWS. - With this method, when you create a connector, in addition to configuring `accessKey` and `secretAccessKey`, you also need to configure `role` and `roleSessionName`. - -4. Create a [Rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-rule.html) in EventBridge. -- The data structure sent to Event Bridge is described [here](## Metadata mapping), and you can create **event pattern** based on this structure. -- Set the target according to your needs. If you're testing this connector, you can set the target to [Cloud Watch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html). - - -### 1. Create a connector - -Depending on the environment, there are several ways to create an AWS EventBridge sink connector: - -- [Create Connector on StreamNative Cloud](https://docs.streamnative.io/docs/connector-create). -- [Create Connector with Function worker](https://pulsar.apache.org/docs/3.0.x/io-quickstart/). - Using this way requires you to download a **NAR** package to create a built-in or non-built-in connector. You can download the version you need from [here](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases). -- [Create Connector with Function mesh](https://functionmesh.io/docs/connectors/run-connector). - Using this way requires you to set the docker image. You can choose the version you want to launch from [here](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge/tags) - -No matter how you create an AWS EventBridge sink connector, the minimum configuration contains the following parameters. - -```json - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } -``` -> * The configuration structure varies depending on how you create the AWS EventBridge sink connector. -> For example, some are **JSON**, some are **YAML**, and some are **Kubernetes YAML**. You need to adapt the configs to the corresponding format. -> -> * If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. - -### 2. Send messages to the topic - -{% callout title="Note" type="note" %} -If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information. -{% /callout %} - -``` java - PulsarClient client = PulsarClient.builder() - .serviceUrl("{{Your Pulsar URL}}") - .build(); - - Producer producer = client.newProducer(Schema.STRING) - .topic("{{Your topic name}}") - .create(); - - String message = "{\"msg\": \"msg-data\"}"; - MessageId msgID = producer.send(message); - System.out.println("Publish " + message + " and message ID " + msgID); - - producer.flush(); - producer.close(); - client.close(); -``` - -### 3. Show data on AWS EventBridge -The connector will send the following format of JSON event to EventBridge. -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail-type": "{{Your topic name}}", - "source": "{{Your connector name}}", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "msg": "msg-data" - }, - "message_id": "124:191:0" - } -} -``` - -## Configuration Properties - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | yes | "" (empty string) | The AWS EventBridge [access key ID.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `secretAccessKey` | String | yes | "" (empty string) | The AWS EventBridge [secret access key.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `region` | String | yes | "" (empty string) | The region where AWS EventBridge service is located. [All AWS region](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/Region.html) | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `role` | String | false | "" (empty string) | The AWS STS [roleArn](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). Example: arn:aws:iam::598203581484:role/test-role | -| `roleSessionName` | String | false | "" (empty string) | The AWS role session name, Name it yourself. | -| `stsEndpoint` | String | false | "" (empty string) | The AWS STS endpoint. By default, the default STS endpoint: https://sts.amazonaws.com is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | false | "" (empty string) | The AWS STS region, By default, the 'region' config or env region is used. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). Example: `arn:aws:events:ap-northeast-1:598263551484:event-bus/my_eventbus` | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -> For details about this connector's advanced features and configurations, see [Advanced features](#advanced-features). - -## Advanced features - -### Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -{% callout title="Note" type="note" %} -Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the support of the Sink downstream system. -{% /callout %} - -### Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -### Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -### Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. - -### Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -### Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -## More Links -- [GitHub Repo](https://github.com/streamnative/pulsar-io-aws-eventbridge) -- [Announcing the Amazon EventBridge Sink Connector for Apache Pulsar](https://streamnative.io/blog/announcing-the-amazon-eventbridge-sink-connector-for-apache-pulsar) -- [Amazon EventBridge connector is now integrated with StreamNative Cloud](https://streamnative.io/blog/amazon-eventbridge-connector-is-now-integrated-with-streamnative-cloud) - - diff --git a/connectors/aws-eventbridge-sink/v3.0.0.2/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v3.0.0.2/aws-eventbridge-sink.md deleted file mode 100644 index c65b87c98..000000000 --- a/connectors/aws-eventbridge-sink/v3.0.0.2/aws-eventbridge-sink.md +++ /dev/null @@ -1,499 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: /images/connectors/aws-eventbridge.png -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: true -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v3.0.0.2/docs/aws-eventbridge-sink.png) - -# Features - -This section describes the features of the AWS EventBridge sink connector. For details about how to configure these -features, see [how to configure](#how-to-configure). - -## Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -{% callout title="Note" type="note" %} -Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the -support of the Sink downstream system. -{% /callout %} - -## Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -## Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -## Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. -> - -## Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -## Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -# How to get - -This section describes how to get the AWS EventBridge sink connector. - -## Work with Function Worker - -You can get the AWS EventBridge sink connector using one of the following methods if you -use [Pulsar Function Worker](https://pulsar.apache.org/docs/en/functions-worker/) to run the connector in a cluster. - -- Download the NAR package - from [the download page](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases/). - -- Build it from the source code. - -To build the AWS EventBridge sink connector from the source code, follow these steps. - -1. Clone the source code to your machine. - - ```bash - git clone https://github.com/streamnative/pulsar-io-aws-eventbridge - ``` - -2. Build the connector in the `pulsar-io-aws-eventbridge` directory. - - ```bash - mvn clean install -DskipTests - ``` - - After the connector is successfully built, a `NAR` package is generated under the target directory. - - ```bash - ls target - pulsar-io-aws-eventbridge-3.0.0.2.nar - ``` - -## Work with Function Mesh - -You can pull the AWS EventBridge sink connector Docker image from -the [Docker Hub](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge) if you -use [Function Mesh](https://functionmesh.io/docs/connectors/run-connector) to run the connector. - -# How to configure - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | No | "" (empty string) | The EventBridge access key ID. | -| `secretAccessKey` | String | no | "" (empty string) | The EventBridge secret access key. | -| `role` | String | no | "" (empty string) | The AWS role to use. | -| `roleSessionName` | String | no | "" (empty string) | The AWS role session name to use. | -| `stsEndpoint` | String | no | "" (empty string) | The STS endpoint to use. By default, the [default STS endpoint](https://sts.amazonaws.com) is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | no | "" (empty string) | The STS region to use, By default, the 'region' config or env region is used. | -| `region` | String | yes | "" (empty string) | The EventBridge region. | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -## Work with Function Worker - -You can create a configuration file (JSON or YAML) to set the properties if you -use [Pulsar Function Worker](https://pulsar.apache.org/docs/en/functions-worker/) to run connectors in a cluster. - -**Example** - -* JSON - - ```json - { - "name": "eventbridge-sink", - "archive": "connectors/pulsar-io-aws-eventbridge-3.0.0.2.nar", - "tenant": "public", - "namespace": "default", - "inputs": [ - "test-aws-eventbridge-pulsar" - ], - "parallelism": 1, - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } - } - ``` - -* YAML - - ```yaml - name: eventbridge-sink - archive: 'connectors/pulsar-io-aws-eventbridge-3.0.0.2.nar' - tenant: public - namespace: default - inputs: - - test-aws-eventbridge-pulsar - parallelism: 1 - configs: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - ``` - -## Work with Function Mesh - -You can create -a [CustomResourceDefinitions (CRD)](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) -to create an AWS EventBridge sink connector. Using CRD makes Function Mesh naturally integrate with the Kubernetes -ecosystem. For more information about Pulsar sink CRD configurations, -see [sink CRD configurations](https://functionmesh.io/docs/connectors/io-crd-config/sink-crd-config). - -You can define a CRD file (YAML) to set the properties as below. - -```yaml -apiVersion: compute.functionmesh.io/v1alpha1 -kind: Sink -metadata: - name: aws-eventbridge-sink-sample -spec: - image: streamnative/pulsar-io-aws-eventbridge:3.0.0.2 - replicas: 1 - maxReplicas: 1 - input: - topics: - - persistent://public/default/test-aws-eventbridge-pulsar - sinkConfig: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - pulsar: - pulsarConfig: "test-pulsar-sink-config" - resources: - limits: - cpu: "0.2" - memory: 1.1G - requests: - cpu: "0.1" - memory: 1G - java: - jar: connectors/pulsar-io-aws-eventbridge-3.0.0.2.jar - clusterName: test-pulsar - autoAck: false -``` - -# How to use - -You can use the AWS EventBridge sink connector with Function Worker or Function Mesh. - -## Work with Function Worker - -{% callout title="Note" type="note" %} -Currently, the AWS EventBridge sink connector cannot run as a built-in connector as it uses the JAR package. -{% /callout %} - -1. Start a Pulsar cluster in standalone mode. - ``` - PULSAR_HOME/bin/pulsar standalone - ``` - -2. Run the AWS EventBridge sink connector. - ``` - PULSAR_HOME/bin/pulsar-admin sinks localrun \ - --sink-config-file - --archive - ``` - - Or, you can create a connector for the Pulsar cluster. - ``` - PULSAR_HOME/bin/pulsar-admin sinks create \ - --sink-config-file - --archive - ``` - -3. Send messages to a Pulsar topic. - - This example sends ten “hello” messages to the `test-aws-eventbridge-pulsar` topic in the `default` namespace of - the `public` tenant. - - ``` - PULSAR_HOME/bin/pulsar-client produce public/default/test-aws-eventbridge-pulsar --messages hello -n 10 - ``` - -4. Show data on AWS EventBridge. - -The connector will send the following format of JSON event to EventBridge. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": "hello" - } -} -``` - -You can configure the [rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-rules.html) to match events on -Amazon EventBridge, and set the target -to [Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html) to track -data. - -## Work with Function Mesh - -This example describes how to create an AWS EventBridge sink connector for a Kubernetes cluster using Function -Mesh. - -### Prerequisites - -- Create and connect to a [Kubernetes cluster](https://kubernetes.io/). - -- Create a [Pulsar cluster](https://pulsar.apache.org/docs/getting-started-helm/) in the Kubernetes cluster. - -- [Install the Function Mesh Operator and CRD](https://functionmesh.io/docs/install-function-mesh/) into the Kubernetes - cluster. - -- Prepare Amazon EventBridge service. For more details, - see [Getting Started with Amazon EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-setup.html). - -### Steps - -1. Define the AWS EventBridge sink connector with a YAML file and save it as `sink-sample.yaml`. - - This example shows how to publish the AWS EventBridge sink connector to Function Mesh with a Docker image. - ```yaml - apiVersion: compute.functionmesh.io/v1alpha1 - kind: Sink - metadata: - name: aws-eventbridge-sink-sample - spec: - image: streamnative/pulsar-io-aws-eventbridge:3.0.0.2 - replicas: 1 - maxReplicas: 1 - input: - topics: - - persistent://public/default/test-aws-eventbridge-pulsar - sinkConfig: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - pulsar: - pulsarConfig: "test-pulsar-sink-config" - resources: - limits: - cpu: "0.2" - memory: 1.1G - requests: - cpu: "0.1" - memory: 1G - java: - jar: connectors/pulsar-io-aws-eventbridge-3.0.0.2.jar - clusterName: test-pulsar - autoAck: false - ``` - -2. Apply the YAML file to create the AWS EventBridge sink connector. - - **Input** - - ``` - kubectl apply -f - ``` - - **Output** - - ``` - sink.compute.functionmesh.io/aws-eventbridge-sink-sample created - ``` - -3. Check whether the AWS EventBridge sink connector is created successfully. - - **Input** - - ``` - kubectl get all - ``` - - **Output** - - ``` - NAME READY STATUS RESTARTS AGE - pod/aws-eventbridge-sink-sample-0 1/1 Running 0 77s - ``` - - After that, you can produce and consume messages using the AWS EventBridge sink connector between Pulsar and - EventBridge. - - - diff --git a/connectors/aws-eventbridge-sink/v3.0.0.3/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v3.0.0.3/aws-eventbridge-sink.md deleted file mode 100644 index fec1c2c3f..000000000 --- a/connectors/aws-eventbridge-sink/v3.0.0.3/aws-eventbridge-sink.md +++ /dev/null @@ -1,498 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: /images/connectors/aws-eventbridge.png -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: true -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v3.0.0.3/docs/aws-eventbridge-sink.png) - -# Features - -This section describes the features of the AWS EventBridge sink connector. For details about how to configure these -features, see [how to configure](#how-to-configure). - -## Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -> **Note:** -> Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the -> support of the Sink downstream system. - -## Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -## Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -## Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. -> - -## Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -## Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -# How to get - -This section describes how to get the AWS EventBridge sink connector. - -## Work with Function Worker - -You can get the AWS EventBridge sink connector using one of the following methods if you -use [Pulsar Function Worker](https://pulsar.apache.org/docs/en/functions-worker/) to run the connector in a cluster. - -- Download the NAR package - from [the download page](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases/). - -- Build it from the source code. - -To build the AWS EventBridge sink connector from the source code, follow these steps. - -1. Clone the source code to your machine. - - ```bash - git clone https://github.com/streamnative/pulsar-io-aws-eventbridge - ``` - -2. Build the connector in the `pulsar-io-aws-eventbridge` directory. - - ```bash - mvn clean install -DskipTests - ``` - - After the connector is successfully built, a `NAR` package is generated under the target directory. - - ```bash - ls target - pulsar-io-aws-eventbridge-3.0.0.3.nar - ``` - -## Work with Function Mesh - -You can pull the AWS EventBridge sink connector Docker image from -the [Docker Hub](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge) if you -use [Function Mesh](https://functionmesh.io/docs/connectors/run-connector) to run the connector. - -# How to configure - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | No | "" (empty string) | The EventBridge access key ID. | -| `secretAccessKey` | String | no | "" (empty string) | The EventBridge secret access key. | -| `role` | String | no | "" (empty string) | The AWS role to use. | -| `roleSessionName` | String | no | "" (empty string) | The AWS role session name to use. | -| `stsEndpoint` | String | no | "" (empty string) | The STS endpoint to use. By default, the [default STS endpoint](https://sts.amazonaws.com) is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | no | "" (empty string) | The STS region to use, By default, the 'region' config or env region is used. | -| `region` | String | yes | "" (empty string) | The EventBridge region. | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -## Work with Function Worker - -You can create a configuration file (JSON or YAML) to set the properties if you -use [Pulsar Function Worker](https://pulsar.apache.org/docs/en/functions-worker/) to run connectors in a cluster. - -**Example** - -* JSON - - ```json - { - "name": "eventbridge-sink", - "archive": "connectors/pulsar-io-aws-eventbridge-3.0.0.3.nar", - "tenant": "public", - "namespace": "default", - "inputs": [ - "test-aws-eventbridge-pulsar" - ], - "parallelism": 1, - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } - } - ``` - -* YAML - - ```yaml - name: eventbridge-sink - archive: 'connectors/pulsar-io-aws-eventbridge-3.0.0.3.nar' - tenant: public - namespace: default - inputs: - - test-aws-eventbridge-pulsar - parallelism: 1 - configs: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - ``` - -## Work with Function Mesh - -You can create -a [CustomResourceDefinitions (CRD)](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) -to create an AWS EventBridge sink connector. Using CRD makes Function Mesh naturally integrate with the Kubernetes -ecosystem. For more information about Pulsar sink CRD configurations, -see [sink CRD configurations](https://functionmesh.io/docs/connectors/io-crd-config/sink-crd-config). - -You can define a CRD file (YAML) to set the properties as below. - -```yaml -apiVersion: compute.functionmesh.io/v1alpha1 -kind: Sink -metadata: - name: aws-eventbridge-sink-sample -spec: - image: streamnative/pulsar-io-aws-eventbridge:3.0.0.3 - replicas: 1 - maxReplicas: 1 - input: - topics: - - persistent://public/default/test-aws-eventbridge-pulsar - sinkConfig: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - pulsar: - pulsarConfig: "test-pulsar-sink-config" - resources: - limits: - cpu: "0.2" - memory: 1.1G - requests: - cpu: "0.1" - memory: 1G - java: - jar: connectors/pulsar-io-aws-eventbridge-3.0.0.3.jar - clusterName: test-pulsar - autoAck: false -``` - -# How to use - -You can use the AWS EventBridge sink connector with Function Worker or Function Mesh. - -## Work with Function Worker - -> **Note** -> -> Currently, the AWS EventBridge sink connector cannot run as a built-in connector as it uses the JAR package. - -1. Start a Pulsar cluster in standalone mode. - ``` - PULSAR_HOME/bin/pulsar standalone - ``` - -2. Run the AWS EventBridge sink connector. - ``` - PULSAR_HOME/bin/pulsar-admin sinks localrun \ - --sink-config-file - --archive - ``` - - Or, you can create a connector for the Pulsar cluster. - ``` - PULSAR_HOME/bin/pulsar-admin sinks create \ - --sink-config-file - --archive - ``` - -3. Send messages to a Pulsar topic. - - This example sends ten “hello” messages to the `test-aws-eventbridge-pulsar` topic in the `default` namespace of - the `public` tenant. - - ``` - PULSAR_HOME/bin/pulsar-client produce public/default/test-aws-eventbridge-pulsar --messages hello -n 10 - ``` - -4. Show data on AWS EventBridge. - -The connector will send the following format of JSON event to EventBridge. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": "hello" - } -} -``` - -You can configure the [rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-rules.html) to match events on -Amazon EventBridge, and set the target -to [Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html) to track -data. - -## Work with Function Mesh - -This example describes how to create an AWS EventBridge sink connector for a Kubernetes cluster using Function -Mesh. - -### Prerequisites - -- Create and connect to a [Kubernetes cluster](https://kubernetes.io/). - -- Create a [Pulsar cluster](https://pulsar.apache.org/docs/getting-started-helm/) in the Kubernetes cluster. - -- [Install the Function Mesh Operator and CRD](https://functionmesh.io/docs/install-function-mesh/) into the Kubernetes - cluster. - -- Prepare Amazon EventBridge service. For more details, - see [Getting Started with Amazon EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-setup.html). - -### Steps - -1. Define the AWS EventBridge sink connector with a YAML file and save it as `sink-sample.yaml`. - - This example shows how to publish the AWS EventBridge sink connector to Function Mesh with a Docker image. - ```yaml - apiVersion: compute.functionmesh.io/v1alpha1 - kind: Sink - metadata: - name: aws-eventbridge-sink-sample - spec: - image: streamnative/pulsar-io-aws-eventbridge:3.0.0.3 - replicas: 1 - maxReplicas: 1 - input: - topics: - - persistent://public/default/test-aws-eventbridge-pulsar - sinkConfig: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - pulsar: - pulsarConfig: "test-pulsar-sink-config" - resources: - limits: - cpu: "0.2" - memory: 1.1G - requests: - cpu: "0.1" - memory: 1G - java: - jar: connectors/pulsar-io-aws-eventbridge-3.0.0.3.jar - clusterName: test-pulsar - autoAck: false - ``` - -2. Apply the YAML file to create the AWS EventBridge sink connector. - - **Input** - - ``` - kubectl apply -f - ``` - - **Output** - - ``` - sink.compute.functionmesh.io/aws-eventbridge-sink-sample created - ``` - -3. Check whether the AWS EventBridge sink connector is created successfully. - - **Input** - - ``` - kubectl get all - ``` - - **Output** - - ``` - NAME READY STATUS RESTARTS AGE - pod/aws-eventbridge-sink-sample-0 1/1 Running 0 77s - ``` - - After that, you can produce and consume messages using the AWS EventBridge sink connector between Pulsar and - EventBridge. - - - diff --git a/connectors/aws-eventbridge-sink/v3.0.0.4/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v3.0.0.4/aws-eventbridge-sink.md deleted file mode 100644 index b1539c6d3..000000000 --- a/connectors/aws-eventbridge-sink/v3.0.0.4/aws-eventbridge-sink.md +++ /dev/null @@ -1,498 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: /images/connectors/aws-eventbridge.png -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: true -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v3.0.0.4/docs/aws-eventbridge-sink.png) - -# Features - -This section describes the features of the AWS EventBridge sink connector. For details about how to configure these -features, see [how to configure](#how-to-configure). - -## Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -> **Note:** -> Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the -> support of the Sink downstream system. - -## Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -## Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -## Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. -> - -## Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -## Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -# How to get - -This section describes how to get the AWS EventBridge sink connector. - -## Work with Function Worker - -You can get the AWS EventBridge sink connector using one of the following methods if you -use [Pulsar Function Worker](https://pulsar.apache.org/docs/en/functions-worker/) to run the connector in a cluster. - -- Download the NAR package - from [the download page](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases/). - -- Build it from the source code. - -To build the AWS EventBridge sink connector from the source code, follow these steps. - -1. Clone the source code to your machine. - - ```bash - git clone https://github.com/streamnative/pulsar-io-aws-eventbridge - ``` - -2. Build the connector in the `pulsar-io-aws-eventbridge` directory. - - ```bash - mvn clean install -DskipTests - ``` - - After the connector is successfully built, a `NAR` package is generated under the target directory. - - ```bash - ls target - pulsar-io-aws-eventbridge-3.0.0.4.nar - ``` - -## Work with Function Mesh - -You can pull the AWS EventBridge sink connector Docker image from -the [Docker Hub](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge) if you -use [Function Mesh](https://functionmesh.io/docs/connectors/run-connector) to run the connector. - -# How to configure - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | No | "" (empty string) | The EventBridge access key ID. | -| `secretAccessKey` | String | no | "" (empty string) | The EventBridge secret access key. | -| `role` | String | no | "" (empty string) | The AWS role to use. | -| `roleSessionName` | String | no | "" (empty string) | The AWS role session name to use. | -| `stsEndpoint` | String | no | "" (empty string) | The STS endpoint to use. By default, the [default STS endpoint](https://sts.amazonaws.com) is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | no | "" (empty string) | The STS region to use, By default, the 'region' config or env region is used. | -| `region` | String | yes | "" (empty string) | The EventBridge region. | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -## Work with Function Worker - -You can create a configuration file (JSON or YAML) to set the properties if you -use [Pulsar Function Worker](https://pulsar.apache.org/docs/en/functions-worker/) to run connectors in a cluster. - -**Example** - -* JSON - - ```json - { - "name": "eventbridge-sink", - "archive": "connectors/pulsar-io-aws-eventbridge-3.0.0.4.nar", - "tenant": "public", - "namespace": "default", - "inputs": [ - "test-aws-eventbridge-pulsar" - ], - "parallelism": 1, - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } - } - ``` - -* YAML - - ```yaml - name: eventbridge-sink - archive: 'connectors/pulsar-io-aws-eventbridge-3.0.0.4.nar' - tenant: public - namespace: default - inputs: - - test-aws-eventbridge-pulsar - parallelism: 1 - configs: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - ``` - -## Work with Function Mesh - -You can create -a [CustomResourceDefinitions (CRD)](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) -to create an AWS EventBridge sink connector. Using CRD makes Function Mesh naturally integrate with the Kubernetes -ecosystem. For more information about Pulsar sink CRD configurations, -see [sink CRD configurations](https://functionmesh.io/docs/connectors/io-crd-config/sink-crd-config). - -You can define a CRD file (YAML) to set the properties as below. - -```yaml -apiVersion: compute.functionmesh.io/v1alpha1 -kind: Sink -metadata: - name: aws-eventbridge-sink-sample -spec: - image: streamnative/pulsar-io-aws-eventbridge:3.0.0.4 - replicas: 1 - maxReplicas: 1 - input: - topics: - - persistent://public/default/test-aws-eventbridge-pulsar - sinkConfig: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - pulsar: - pulsarConfig: "test-pulsar-sink-config" - resources: - limits: - cpu: "0.2" - memory: 1.1G - requests: - cpu: "0.1" - memory: 1G - java: - jar: connectors/pulsar-io-aws-eventbridge-3.0.0.4.jar - clusterName: test-pulsar - autoAck: false -``` - -# How to use - -You can use the AWS EventBridge sink connector with Function Worker or Function Mesh. - -## Work with Function Worker - -> **Note** -> -> Currently, the AWS EventBridge sink connector cannot run as a built-in connector as it uses the JAR package. - -1. Start a Pulsar cluster in standalone mode. - ``` - PULSAR_HOME/bin/pulsar standalone - ``` - -2. Run the AWS EventBridge sink connector. - ``` - PULSAR_HOME/bin/pulsar-admin sinks localrun \ - --sink-config-file - --archive - ``` - - Or, you can create a connector for the Pulsar cluster. - ``` - PULSAR_HOME/bin/pulsar-admin sinks create \ - --sink-config-file - --archive - ``` - -3. Send messages to a Pulsar topic. - - This example sends ten “hello” messages to the `test-aws-eventbridge-pulsar` topic in the `default` namespace of - the `public` tenant. - - ``` - PULSAR_HOME/bin/pulsar-client produce public/default/test-aws-eventbridge-pulsar --messages hello -n 10 - ``` - -4. Show data on AWS EventBridge. - -The connector will send the following format of JSON event to EventBridge. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": "hello" - } -} -``` - -You can configure the [rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-rules.html) to match events on -Amazon EventBridge, and set the target -to [Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html) to track -data. - -## Work with Function Mesh - -This example describes how to create an AWS EventBridge sink connector for a Kubernetes cluster using Function -Mesh. - -### Prerequisites - -- Create and connect to a [Kubernetes cluster](https://kubernetes.io/). - -- Create a [Pulsar cluster](https://pulsar.apache.org/docs/getting-started-helm/) in the Kubernetes cluster. - -- [Install the Function Mesh Operator and CRD](https://functionmesh.io/docs/install-function-mesh/) into the Kubernetes - cluster. - -- Prepare Amazon EventBridge service. For more details, - see [Getting Started with Amazon EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-setup.html). - -### Steps - -1. Define the AWS EventBridge sink connector with a YAML file and save it as `sink-sample.yaml`. - - This example shows how to publish the AWS EventBridge sink connector to Function Mesh with a Docker image. - ```yaml - apiVersion: compute.functionmesh.io/v1alpha1 - kind: Sink - metadata: - name: aws-eventbridge-sink-sample - spec: - image: streamnative/pulsar-io-aws-eventbridge:3.0.0.4 - replicas: 1 - maxReplicas: 1 - input: - topics: - - persistent://public/default/test-aws-eventbridge-pulsar - sinkConfig: - accessKeyId: '{{Your access access key}}' - secretAccessKey: '{{Your secret access key}}' - region: test-region - eventBusName: test-event-bus-name - pulsar: - pulsarConfig: "test-pulsar-sink-config" - resources: - limits: - cpu: "0.2" - memory: 1.1G - requests: - cpu: "0.1" - memory: 1G - java: - jar: connectors/pulsar-io-aws-eventbridge-3.0.0.4.jar - clusterName: test-pulsar - autoAck: false - ``` - -2. Apply the YAML file to create the AWS EventBridge sink connector. - - **Input** - - ``` - kubectl apply -f - ``` - - **Output** - - ``` - sink.compute.functionmesh.io/aws-eventbridge-sink-sample created - ``` - -3. Check whether the AWS EventBridge sink connector is created successfully. - - **Input** - - ``` - kubectl get all - ``` - - **Output** - - ``` - NAME READY STATUS RESTARTS AGE - pod/aws-eventbridge-sink-sample-0 1/1 Running 0 77s - ``` - - After that, you can produce and consume messages using the AWS EventBridge sink connector between Pulsar and - EventBridge. - - - diff --git a/connectors/aws-eventbridge-sink/v3.0.1.1/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v3.0.1.1/aws-eventbridge-sink.md deleted file mode 100644 index a3d6e91d4..000000000 --- a/connectors/aws-eventbridge-sink/v3.0.1.1/aws-eventbridge-sink.md +++ /dev/null @@ -1,343 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: "/images/connectors/aws-eventbridge.png" -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: "true" -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v3.0.1.1/docs/aws-eventbridge-sink.png) - -## Quick start - -### Prerequisites - -The prerequisites for connecting an AWS EventBridge sink connector to external systems include: - -1. Create EventBridge and EventBus in AWS. -2. Create the [AWS User](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) and create `AccessKey`(Please record `AccessKey` and `AecretAccessKey`). -3. Assign permissions to Pulsar users, and ensure they have the following permissions to the AWS EventBus. For details, see [permissions for event buses](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-bus-perms.html) - -```json - { - "Version": "2012-10-17", - "Statement": [ - - ### To grant permissions for an account to use the PutEvents action, include the following, otherwise delete this section: ### - - { - - "Sid": "AllowAccountToPutEvents", - "Effect": "Allow", - "Principal": { - "AWS": "" - }, - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - }, - - ### Include the following section to grant permissions to all members of your AWS Organizations to use the PutEvents action ### - - { - "Sid": "AllowAllAccountsFromOrganizationToPutEvents", - "Effect": "Allow", - "Principal": "*", - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - "Condition": { - "StringEquals": { - "aws:PrincipalOrgID": "o-yourOrgID" - } - } - } - ] -} -``` -- You can set permissions directly for this user. With this method, when you create a connector, you only need to configure `accessKey` and `secretAccessKey`. -- Or you can use [Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html), this [video](https://www.youtube.com/watch?v=dqF4VJCska4) explains how to use STS on AWS. - With this method, when you create a connector, in addition to configuring `accessKey` and `secretAccessKey`, you also need to configure `role` and `roleSessionName`. - -4. Create a [Rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-rule.html) in EventBridge. -- The data structure sent to Event Bridge is described [here](## Metadata mapping), and you can create **event pattern** based on this structure. -- Set the target according to your needs. If you're testing this connector, you can set the target to [Cloud Watch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html). - - -### 1. Create a connector - -Depending on the environment, there are several ways to create an AWS EventBridge sink connector: - -- [Create Connector on StreamNative Cloud](https://docs.streamnative.io/docs/connector-create). -- [Create Connector with Function worker](https://pulsar.apache.org/docs/3.0.x/io-quickstart/). - Using this way requires you to download a **NAR** package to create a built-in or non-built-in connector. You can download the version you need from [here](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases). -- [Create Connector with Function mesh](https://functionmesh.io/docs/connectors/run-connector). - Using this way requires you to set the docker image. You can choose the version you want to launch from [here](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge/tags) - -No matter how you create an AWS EventBridge sink connector, the minimum configuration contains the following parameters. - -```json - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } -``` -> * The configuration structure varies depending on how you create the AWS EventBridge sink connector. -> For example, some are **JSON**, some are **YAML**, and some are **Kubernetes YAML**. You need to adapt the configs to the corresponding format. -> -> * If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. - -### 2. Send messages to the topic -> **Note:** If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information. - -``` java - PulsarClient client = PulsarClient.builder() - .serviceUrl("{{Your Pulsar URL}}") - .build(); - - Producer producer = client.newProducer(Schema.STRING) - .topic("{{Your topic name}}") - .create(); - - String message = "{\"msg\": \"msg-data\"}"; - MessageId msgID = producer.send(message); - System.out.println("Publish " + message + " and message ID " + msgID); - - producer.flush(); - producer.close(); - client.close(); -``` - -### 3. Show data on AWS EventBridge -The connector will send the following format of JSON event to EventBridge. -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail-type": "{{Your topic name}}", - "source": "{{Your connector name}}", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "msg": "msg-data" - }, - "message_id": "124:191:0" - } -} -``` - -## Configuration Properties - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | yes | "" (empty string) | The AWS EventBridge [access key ID.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `secretAccessKey` | String | yes | "" (empty string) | The AWS EventBridge [secret access key.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `region` | String | yes | "" (empty string) | The region where AWS EventBridge service is located. [All AWS region](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/Region.html) | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `role` | String | false | "" (empty string) | The AWS STS [roleArn](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). Example: arn:aws:iam::598203581484:role/test-role | -| `roleSessionName` | String | false | "" (empty string) | The AWS role session name, Name it yourself. | -| `stsEndpoint` | String | false | "" (empty string) | The AWS STS endpoint. By default, the default STS endpoint: https://sts.amazonaws.com is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | false | "" (empty string) | The AWS STS region, By default, the 'region' config or env region is used. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). Example: `arn:aws:events:ap-northeast-1:598263551484:event-bus/my_eventbus` | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -> For details about this connector's advanced features and configurations, see [Advanced features](#advanced-features). - -## Advanced features - -### Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -> **Note:** -> Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the -> support of the Sink downstream system. - -### Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -### Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -### Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. - -### Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -### Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -## More Links -- [GitHub Repo](https://github.com/streamnative/pulsar-io-aws-eventbridge) -- [Announcing the Amazon EventBridge Sink Connector for Apache Pulsar](https://streamnative.io/blog/announcing-the-amazon-eventbridge-sink-connector-for-apache-pulsar) -- [Amazon EventBridge connector is now integrated with StreamNative Cloud](https://streamnative.io/blog/amazon-eventbridge-connector-is-now-integrated-with-streamnative-cloud) - - diff --git a/connectors/aws-eventbridge-sink/v3.0.1.2/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v3.0.1.2/aws-eventbridge-sink.md deleted file mode 100644 index 8381c5572..000000000 --- a/connectors/aws-eventbridge-sink/v3.0.1.2/aws-eventbridge-sink.md +++ /dev/null @@ -1,346 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: "/images/connectors/aws-eventbridge.png" -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: "true" -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v3.0.1.2/docs/aws-eventbridge-sink.png) - -## Quick start - -### Prerequisites - -The prerequisites for connecting an AWS EventBridge sink connector to external systems include: - -1. Create EventBridge and EventBus in AWS. -2. Create the [AWS User](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) and create `AccessKey`(Please record `AccessKey` and `SecretAccessKey`). -3. Assign permissions to AWS User, and ensure they have the following permissions to the AWS EventBus. For details, see [permissions for event buses](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-bus-perms.html) - -```json - { - "Version": "2012-10-17", - "Statement": [ - - ### To grant permissions for an account to use the PutEvents action, include the following, otherwise delete this section: ### - - { - - "Sid": "AllowAccountToPutEvents", - "Effect": "Allow", - "Principal": { - "AWS": "" - }, - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - }, - - ### Include the following section to grant permissions to all members of your AWS Organizations to use the PutEvents action ### - - { - "Sid": "AllowAllAccountsFromOrganizationToPutEvents", - "Effect": "Allow", - "Principal": "*", - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - "Condition": { - "StringEquals": { - "aws:PrincipalOrgID": "o-yourOrgID" - } - } - } - ] -} -``` -- You can set permissions directly for this user. With this method, when you create a connector, you only need to configure `accessKey` and `secretAccessKey`. -- Or you can use [Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html), this [video](https://www.youtube.com/watch?v=dqF4VJCska4) explains how to use STS on AWS. - With this method, when you create a connector, in addition to configuring `accessKey` and `secretAccessKey`, you also need to configure `role` and `roleSessionName`. - -4. Create a [Rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-rule.html) in EventBridge. -- The data structure sent to Event Bridge is described [here](## Metadata mapping), and you can create **event pattern** based on this structure. -- Set the target according to your needs. If you're testing this connector, you can set the target to [Cloud Watch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html). - - -### 1. Create a connector - -Depending on the environment, there are several ways to create an AWS EventBridge sink connector: - -- [Create Connector on StreamNative Cloud](https://docs.streamnative.io/docs/connector-create). -- [Create Connector with Function worker](https://pulsar.apache.org/docs/3.0.x/io-quickstart/). - Using this way requires you to download a **NAR** package to create a built-in or non-built-in connector. You can download the version you need from [here](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases). -- [Create Connector with Function mesh](https://functionmesh.io/docs/connectors/run-connector). - Using this way requires you to set the docker image. You can choose the version you want to launch from [here](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge/tags) - -No matter how you create an AWS EventBridge sink connector, the minimum configuration contains the following parameters. - -```json - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } -``` -> * The configuration structure varies depending on how you create the AWS EventBridge sink connector. -> For example, some are **JSON**, some are **YAML**, and some are **Kubernetes YAML**. You need to adapt the configs to the corresponding format. -> -> * If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. - -### 2. Send messages to the topic - -{% callout title="Note" type="note" %} -If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information. -{% /callout %} - -``` java - PulsarClient client = PulsarClient.builder() - .serviceUrl("{{Your Pulsar URL}}") - .build(); - - Producer producer = client.newProducer(Schema.STRING) - .topic("{{Your topic name}}") - .create(); - - String message = "{\"msg\": \"msg-data\"}"; - MessageId msgID = producer.send(message); - System.out.println("Publish " + message + " and message ID " + msgID); - - producer.flush(); - producer.close(); - client.close(); -``` - -### 3. Show data on AWS EventBridge -The connector will send the following format of JSON event to EventBridge. -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail-type": "{{Your topic name}}", - "source": "{{Your connector name}}", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "msg": "msg-data" - }, - "message_id": "124:191:0" - } -} -``` - -## Configuration Properties - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | yes | "" (empty string) | The AWS EventBridge [access key ID.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `secretAccessKey` | String | yes | "" (empty string) | The AWS EventBridge [secret access key.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `region` | String | yes | "" (empty string) | The region where AWS EventBridge service is located. [All AWS region](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/Region.html) | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `role` | String | false | "" (empty string) | The AWS STS [roleArn](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). Example: arn:aws:iam::598203581484:role/test-role | -| `roleSessionName` | String | false | "" (empty string) | The AWS role session name, Name it yourself. | -| `stsEndpoint` | String | false | "" (empty string) | The AWS STS endpoint. By default, the default STS endpoint: https://sts.amazonaws.com is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | false | "" (empty string) | The AWS STS region, By default, the 'region' config or env region is used. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). Example: `arn:aws:events:ap-northeast-1:598263551484:event-bus/my_eventbus` | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -> For details about this connector's advanced features and configurations, see [Advanced features](#advanced-features). - -## Advanced features - -### Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -{% callout title="Note" type="note" %} -Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the support of the Sink downstream system. -{% /callout %} - -### Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -### Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -### Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. - -### Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -### Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -## More Links -- [GitHub Repo](https://github.com/streamnative/pulsar-io-aws-eventbridge) -- [Announcing the Amazon EventBridge Sink Connector for Apache Pulsar](https://streamnative.io/blog/announcing-the-amazon-eventbridge-sink-connector-for-apache-pulsar) -- [Amazon EventBridge connector is now integrated with StreamNative Cloud](https://streamnative.io/blog/amazon-eventbridge-connector-is-now-integrated-with-streamnative-cloud) - - diff --git a/connectors/aws-eventbridge-sink/v3.0.1.3/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v3.0.1.3/aws-eventbridge-sink.md deleted file mode 100644 index b3b9603dc..000000000 --- a/connectors/aws-eventbridge-sink/v3.0.1.3/aws-eventbridge-sink.md +++ /dev/null @@ -1,346 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: "/images/connectors/aws-eventbridge.png" -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: "true" -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v3.0.1.3/docs/aws-eventbridge-sink.png) - -## Quick start - -### Prerequisites - -The prerequisites for connecting an AWS EventBridge sink connector to external systems include: - -1. Create EventBridge and EventBus in AWS. -2. Create the [AWS User](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) and create `AccessKey`(Please record `AccessKey` and `SecretAccessKey`). -3. Assign permissions to AWS User, and ensure they have the following permissions to the AWS EventBus. For details, see [permissions for event buses](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-bus-perms.html) - -```json - { - "Version": "2012-10-17", - "Statement": [ - - ### To grant permissions for an account to use the PutEvents action, include the following, otherwise delete this section: ### - - { - - "Sid": "AllowAccountToPutEvents", - "Effect": "Allow", - "Principal": { - "AWS": "" - }, - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - }, - - ### Include the following section to grant permissions to all members of your AWS Organizations to use the PutEvents action ### - - { - "Sid": "AllowAllAccountsFromOrganizationToPutEvents", - "Effect": "Allow", - "Principal": "*", - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - "Condition": { - "StringEquals": { - "aws:PrincipalOrgID": "o-yourOrgID" - } - } - } - ] -} -``` -- You can set permissions directly for this user. With this method, when you create a connector, you only need to configure `accessKey` and `secretAccessKey`. -- Or you can use [Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html), this [video](https://www.youtube.com/watch?v=dqF4VJCska4) explains how to use STS on AWS. - With this method, when you create a connector, in addition to configuring `accessKey` and `secretAccessKey`, you also need to configure `role` and `roleSessionName`. - -4. Create a [Rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-rule.html) in EventBridge. -- The data structure sent to Event Bridge is described [here](## Metadata mapping), and you can create **event pattern** based on this structure. -- Set the target according to your needs. If you're testing this connector, you can set the target to [Cloud Watch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html). - - -### 1. Create a connector - -Depending on the environment, there are several ways to create an AWS EventBridge sink connector: - -- [Create Connector on StreamNative Cloud](https://docs.streamnative.io/docs/connector-create). -- [Create Connector with Function worker](https://pulsar.apache.org/docs/3.0.x/io-quickstart/). - Using this way requires you to download a **NAR** package to create a built-in or non-built-in connector. You can download the version you need from [here](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases). -- [Create Connector with Function mesh](https://functionmesh.io/docs/connectors/run-connector). - Using this way requires you to set the docker image. You can choose the version you want to launch from [here](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge/tags) - -No matter how you create an AWS EventBridge sink connector, the minimum configuration contains the following parameters. - -```json - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } -``` -> * The configuration structure varies depending on how you create the AWS EventBridge sink connector. -> For example, some are **JSON**, some are **YAML**, and some are **Kubernetes YAML**. You need to adapt the configs to the corresponding format. -> -> * If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. - -### 2. Send messages to the topic - -{% callout title="Note" type="note" %} -If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information. -{% /callout %} - -``` java - PulsarClient client = PulsarClient.builder() - .serviceUrl("{{Your Pulsar URL}}") - .build(); - - Producer producer = client.newProducer(Schema.STRING) - .topic("{{Your topic name}}") - .create(); - - String message = "{\"msg\": \"msg-data\"}"; - MessageId msgID = producer.send(message); - System.out.println("Publish " + message + " and message ID " + msgID); - - producer.flush(); - producer.close(); - client.close(); -``` - -### 3. Show data on AWS EventBridge -The connector will send the following format of JSON event to EventBridge. -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail-type": "{{Your topic name}}", - "source": "{{Your connector name}}", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "msg": "msg-data" - }, - "message_id": "124:191:0" - } -} -``` - -## Configuration Properties - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | yes | "" (empty string) | The AWS EventBridge [access key ID.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `secretAccessKey` | String | yes | "" (empty string) | The AWS EventBridge [secret access key.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `region` | String | yes | "" (empty string) | The region where AWS EventBridge service is located. [All AWS region](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/Region.html) | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `role` | String | false | "" (empty string) | The AWS STS [roleArn](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). Example: arn:aws:iam::598203581484:role/test-role | -| `roleSessionName` | String | false | "" (empty string) | The AWS role session name, Name it yourself. | -| `stsEndpoint` | String | false | "" (empty string) | The AWS STS endpoint. By default, the default STS endpoint: https://sts.amazonaws.com is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | false | "" (empty string) | The AWS STS region, By default, the 'region' config or env region is used. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). Example: `arn:aws:events:ap-northeast-1:598263551484:event-bus/my_eventbus` | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -> For details about this connector's advanced features and configurations, see [Advanced features](#advanced-features). - -## Advanced features - -### Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -{% callout title="Note" type="note" %} -Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the support of the Sink downstream system. -{% /callout %} - -### Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -### Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -### Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. - -### Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -### Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -## More Links -- [GitHub Repo](https://github.com/streamnative/pulsar-io-aws-eventbridge) -- [Announcing the Amazon EventBridge Sink Connector for Apache Pulsar](https://streamnative.io/blog/announcing-the-amazon-eventbridge-sink-connector-for-apache-pulsar) -- [Amazon EventBridge connector is now integrated with StreamNative Cloud](https://streamnative.io/blog/amazon-eventbridge-connector-is-now-integrated-with-streamnative-cloud) - - diff --git a/connectors/aws-eventbridge-sink/v3.0.1.4/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v3.0.1.4/aws-eventbridge-sink.md deleted file mode 100644 index 46ec0ba6b..000000000 --- a/connectors/aws-eventbridge-sink/v3.0.1.4/aws-eventbridge-sink.md +++ /dev/null @@ -1,346 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: "/images/connectors/aws-eventbridge.png" -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: "true" -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v3.0.1.4/docs/aws-eventbridge-sink.png) - -## Quick start - -### Prerequisites - -The prerequisites for connecting an AWS EventBridge sink connector to external systems include: - -1. Create EventBridge and EventBus in AWS. -2. Create the [AWS User](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) and create `AccessKey`(Please record `AccessKey` and `SecretAccessKey`). -3. Assign permissions to AWS User, and ensure they have the following permissions to the AWS EventBus. For details, see [permissions for event buses](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-bus-perms.html) - -```json - { - "Version": "2012-10-17", - "Statement": [ - - ### To grant permissions for an account to use the PutEvents action, include the following, otherwise delete this section: ### - - { - - "Sid": "AllowAccountToPutEvents", - "Effect": "Allow", - "Principal": { - "AWS": "" - }, - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - }, - - ### Include the following section to grant permissions to all members of your AWS Organizations to use the PutEvents action ### - - { - "Sid": "AllowAllAccountsFromOrganizationToPutEvents", - "Effect": "Allow", - "Principal": "*", - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - "Condition": { - "StringEquals": { - "aws:PrincipalOrgID": "o-yourOrgID" - } - } - } - ] -} -``` -- You can set permissions directly for this user. With this method, when you create a connector, you only need to configure `accessKey` and `secretAccessKey`. -- Or you can use [Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html), this [video](https://www.youtube.com/watch?v=dqF4VJCska4) explains how to use STS on AWS. - With this method, when you create a connector, in addition to configuring `accessKey` and `secretAccessKey`, you also need to configure `role` and `roleSessionName`. - -4. Create a [Rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-rule.html) in EventBridge. -- The data structure sent to Event Bridge is described [here](## Metadata mapping), and you can create **event pattern** based on this structure. -- Set the target according to your needs. If you're testing this connector, you can set the target to [Cloud Watch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html). - - -### 1. Create a connector - -Depending on the environment, there are several ways to create an AWS EventBridge sink connector: - -- [Create Connector on StreamNative Cloud](https://docs.streamnative.io/docs/connector-create). -- [Create Connector with Function worker](https://pulsar.apache.org/docs/3.0.x/io-quickstart/). - Using this way requires you to download a **NAR** package to create a built-in or non-built-in connector. You can download the version you need from [here](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases). -- [Create Connector with Function mesh](https://functionmesh.io/docs/connectors/run-connector). - Using this way requires you to set the docker image. You can choose the version you want to launch from [here](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge/tags) - -No matter how you create an AWS EventBridge sink connector, the minimum configuration contains the following parameters. - -```json - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } -``` -> * The configuration structure varies depending on how you create the AWS EventBridge sink connector. -> For example, some are **JSON**, some are **YAML**, and some are **Kubernetes YAML**. You need to adapt the configs to the corresponding format. -> -> * If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. - -### 2. Send messages to the topic - -{% callout title="Note" type="note" %} -If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information. -{% /callout %} - -``` java - PulsarClient client = PulsarClient.builder() - .serviceUrl("{{Your Pulsar URL}}") - .build(); - - Producer producer = client.newProducer(Schema.STRING) - .topic("{{Your topic name}}") - .create(); - - String message = "{\"msg\": \"msg-data\"}"; - MessageId msgID = producer.send(message); - System.out.println("Publish " + message + " and message ID " + msgID); - - producer.flush(); - producer.close(); - client.close(); -``` - -### 3. Show data on AWS EventBridge -The connector will send the following format of JSON event to EventBridge. -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail-type": "{{Your topic name}}", - "source": "{{Your connector name}}", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "msg": "msg-data" - }, - "message_id": "124:191:0" - } -} -``` - -## Configuration Properties - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | yes | "" (empty string) | The AWS EventBridge [access key ID.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `secretAccessKey` | String | yes | "" (empty string) | The AWS EventBridge [secret access key.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `region` | String | yes | "" (empty string) | The region where AWS EventBridge service is located. [All AWS region](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/Region.html) | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `role` | String | false | "" (empty string) | The AWS STS [roleArn](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). Example: arn:aws:iam::598203581484:role/test-role | -| `roleSessionName` | String | false | "" (empty string) | The AWS role session name, Name it yourself. | -| `stsEndpoint` | String | false | "" (empty string) | The AWS STS endpoint. By default, the default STS endpoint: https://sts.amazonaws.com is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | false | "" (empty string) | The AWS STS region, By default, the 'region' config or env region is used. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). Example: `arn:aws:events:ap-northeast-1:598263551484:event-bus/my_eventbus` | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -> For details about this connector's advanced features and configurations, see [Advanced features](#advanced-features). - -## Advanced features - -### Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -{% callout title="Note" type="note" %} -Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the support of the Sink downstream system. -{% /callout %} - -### Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -### Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -### Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. - -### Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -### Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -## More Links -- [GitHub Repo](https://github.com/streamnative/pulsar-io-aws-eventbridge) -- [Announcing the Amazon EventBridge Sink Connector for Apache Pulsar](https://streamnative.io/blog/announcing-the-amazon-eventbridge-sink-connector-for-apache-pulsar) -- [Amazon EventBridge connector is now integrated with StreamNative Cloud](https://streamnative.io/blog/amazon-eventbridge-connector-is-now-integrated-with-streamnative-cloud) - - diff --git a/connectors/aws-eventbridge-sink/v3.0.1.5/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v3.0.1.5/aws-eventbridge-sink.md deleted file mode 100644 index b566bf787..000000000 --- a/connectors/aws-eventbridge-sink/v3.0.1.5/aws-eventbridge-sink.md +++ /dev/null @@ -1,346 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: "/images/connectors/aws-eventbridge.png" -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: "true" -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v3.0.1.5/docs/aws-eventbridge-sink.png) - -## Quick start - -### Prerequisites - -The prerequisites for connecting an AWS EventBridge sink connector to external systems include: - -1. Create EventBridge and EventBus in AWS. -2. Create the [AWS User](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) and create `AccessKey`(Please record `AccessKey` and `SecretAccessKey`). -3. Assign permissions to AWS User, and ensure they have the following permissions to the AWS EventBus. For details, see [permissions for event buses](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-bus-perms.html) - -```json - { - "Version": "2012-10-17", - "Statement": [ - - ### To grant permissions for an account to use the PutEvents action, include the following, otherwise delete this section: ### - - { - - "Sid": "AllowAccountToPutEvents", - "Effect": "Allow", - "Principal": { - "AWS": "" - }, - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - }, - - ### Include the following section to grant permissions to all members of your AWS Organizations to use the PutEvents action ### - - { - "Sid": "AllowAllAccountsFromOrganizationToPutEvents", - "Effect": "Allow", - "Principal": "*", - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - "Condition": { - "StringEquals": { - "aws:PrincipalOrgID": "o-yourOrgID" - } - } - } - ] -} -``` -- You can set permissions directly for this user. With this method, when you create a connector, you only need to configure `accessKey` and `secretAccessKey`. -- Or you can use [Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html), this [video](https://www.youtube.com/watch?v=dqF4VJCska4) explains how to use STS on AWS. - With this method, when you create a connector, in addition to configuring `accessKey` and `secretAccessKey`, you also need to configure `role` and `roleSessionName`. - -4. Create a [Rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-rule.html) in EventBridge. -- The data structure sent to Event Bridge is described [here](## Metadata mapping), and you can create **event pattern** based on this structure. -- Set the target according to your needs. If you're testing this connector, you can set the target to [Cloud Watch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html). - - -### 1. Create a connector - -Depending on the environment, there are several ways to create an AWS EventBridge sink connector: - -- [Create Connector on StreamNative Cloud](https://docs.streamnative.io/docs/connector-create). -- [Create Connector with Function worker](https://pulsar.apache.org/docs/3.0.x/io-quickstart/). - Using this way requires you to download a **NAR** package to create a built-in or non-built-in connector. You can download the version you need from [here](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases). -- [Create Connector with Function mesh](https://functionmesh.io/docs/connectors/run-connector). - Using this way requires you to set the docker image. You can choose the version you want to launch from [here](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge/tags) - -No matter how you create an AWS EventBridge sink connector, the minimum configuration contains the following parameters. - -```json - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } -``` -> * The configuration structure varies depending on how you create the AWS EventBridge sink connector. -> For example, some are **JSON**, some are **YAML**, and some are **Kubernetes YAML**. You need to adapt the configs to the corresponding format. -> -> * If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. - -### 2. Send messages to the topic - -{% callout title="Note" type="note" %} -If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information. -{% /callout %} - -``` java - PulsarClient client = PulsarClient.builder() - .serviceUrl("{{Your Pulsar URL}}") - .build(); - - Producer producer = client.newProducer(Schema.STRING) - .topic("{{Your topic name}}") - .create(); - - String message = "{\"msg\": \"msg-data\"}"; - MessageId msgID = producer.send(message); - System.out.println("Publish " + message + " and message ID " + msgID); - - producer.flush(); - producer.close(); - client.close(); -``` - -### 3. Show data on AWS EventBridge -The connector will send the following format of JSON event to EventBridge. -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail-type": "{{Your topic name}}", - "source": "{{Your connector name}}", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "msg": "msg-data" - }, - "message_id": "124:191:0" - } -} -``` - -## Configuration Properties - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | yes | "" (empty string) | The AWS EventBridge [access key ID.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `secretAccessKey` | String | yes | "" (empty string) | The AWS EventBridge [secret access key.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `region` | String | yes | "" (empty string) | The region where AWS EventBridge service is located. [All AWS region](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/Region.html) | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `role` | String | false | "" (empty string) | The AWS STS [roleArn](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). Example: arn:aws:iam::598203581484:role/test-role | -| `roleSessionName` | String | false | "" (empty string) | The AWS role session name, Name it yourself. | -| `stsEndpoint` | String | false | "" (empty string) | The AWS STS endpoint. By default, the default STS endpoint: https://sts.amazonaws.com is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | false | "" (empty string) | The AWS STS region, By default, the 'region' config or env region is used. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). Example: `arn:aws:events:ap-northeast-1:598263551484:event-bus/my_eventbus` | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -> For details about this connector's advanced features and configurations, see [Advanced features](#advanced-features). - -## Advanced features - -### Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -{% callout title="Note" type="note" %} -Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the support of the Sink downstream system. -{% /callout %} - -### Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -### Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -### Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. - -### Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -### Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -## More Links -- [GitHub Repo](https://github.com/streamnative/pulsar-io-aws-eventbridge) -- [Announcing the Amazon EventBridge Sink Connector for Apache Pulsar](https://streamnative.io/blog/announcing-the-amazon-eventbridge-sink-connector-for-apache-pulsar) -- [Amazon EventBridge connector is now integrated with StreamNative Cloud](https://streamnative.io/blog/amazon-eventbridge-connector-is-now-integrated-with-streamnative-cloud) - - diff --git a/connectors/aws-eventbridge-sink/v3.0.1.6/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v3.0.1.6/aws-eventbridge-sink.md index b28ce1d0e..432adc6b6 100644 --- a/connectors/aws-eventbridge-sink/v3.0.1.6/aws-eventbridge-sink.md +++ b/connectors/aws-eventbridge-sink/v3.0.1.6/aws-eventbridge-sink.md @@ -11,13 +11,13 @@ tags: alias: AWS EventBridge Sink Connector features: [""] icon: "/images/connectors/aws-eventbridge.png" -download: +download: Business License support: streamnative -support_link: +support_link: support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" owner_name: "streamnative" owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: +dockerfile: sn_available: "true" id: "aws-eventbridge-sink" --- diff --git a/connectors/aws-eventbridge-sink/v3.1.0.1/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v3.1.0.1/aws-eventbridge-sink.md deleted file mode 100644 index c006fbbc1..000000000 --- a/connectors/aws-eventbridge-sink/v3.1.0.1/aws-eventbridge-sink.md +++ /dev/null @@ -1,343 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: "/images/connectors/aws-eventbridge.png" -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: "true" -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v3.1.0.1/docs/aws-eventbridge-sink.png) - -## Quick start - -### Prerequisites - -The prerequisites for connecting an AWS EventBridge sink connector to external systems include: - -1. Create EventBridge and EventBus in AWS. -2. Create the [AWS User](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) and create `AccessKey`(Please record `AccessKey` and `AecretAccessKey`). -3. Assign permissions to Pulsar users, and ensure they have the following permissions to the AWS EventBus. For details, see [permissions for event buses](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-bus-perms.html) - -```json - { - "Version": "2012-10-17", - "Statement": [ - - ### To grant permissions for an account to use the PutEvents action, include the following, otherwise delete this section: ### - - { - - "Sid": "AllowAccountToPutEvents", - "Effect": "Allow", - "Principal": { - "AWS": "" - }, - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - }, - - ### Include the following section to grant permissions to all members of your AWS Organizations to use the PutEvents action ### - - { - "Sid": "AllowAllAccountsFromOrganizationToPutEvents", - "Effect": "Allow", - "Principal": "*", - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - "Condition": { - "StringEquals": { - "aws:PrincipalOrgID": "o-yourOrgID" - } - } - } - ] -} -``` -- You can set permissions directly for this user. With this method, when you create a connector, you only need to configure `accessKey` and `secretAccessKey`. -- Or you can use [Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html), this [video](https://www.youtube.com/watch?v=dqF4VJCska4) explains how to use STS on AWS. - With this method, when you create a connector, in addition to configuring `accessKey` and `secretAccessKey`, you also need to configure `role` and `roleSessionName`. - -4. Create a [Rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-rule.html) in EventBridge. -- The data structure sent to Event Bridge is described [here](## Metadata mapping), and you can create **event pattern** based on this structure. -- Set the target according to your needs. If you're testing this connector, you can set the target to [Cloud Watch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html). - - -### 1. Create a connector - -Depending on the environment, there are several ways to create an AWS EventBridge sink connector: - -- [Create Connector on StreamNative Cloud](https://docs.streamnative.io/docs/connector-create). -- [Create Connector with Function worker](https://pulsar.apache.org/docs/3.0.x/io-quickstart/). - Using this way requires you to download a **NAR** package to create a built-in or non-built-in connector. You can download the version you need from [here](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases). -- [Create Connector with Function mesh](https://functionmesh.io/docs/connectors/run-connector). - Using this way requires you to set the docker image. You can choose the version you want to launch from [here](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge/tags) - -No matter how you create an AWS EventBridge sink connector, the minimum configuration contains the following parameters. - -```json - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } -``` -> * The configuration structure varies depending on how you create the AWS EventBridge sink connector. -> For example, some are **JSON**, some are **YAML**, and some are **Kubernetes YAML**. You need to adapt the configs to the corresponding format. -> -> * If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. - -### 2. Send messages to the topic -> **Note:** If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information. - -``` java - PulsarClient client = PulsarClient.builder() - .serviceUrl("{{Your Pulsar URL}}") - .build(); - - Producer producer = client.newProducer(Schema.STRING) - .topic("{{Your topic name}}") - .create(); - - String message = "{\"msg\": \"msg-data\"}"; - MessageId msgID = producer.send(message); - System.out.println("Publish " + message + " and message ID " + msgID); - - producer.flush(); - producer.close(); - client.close(); -``` - -### 3. Show data on AWS EventBridge -The connector will send the following format of JSON event to EventBridge. -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail-type": "{{Your topic name}}", - "source": "{{Your connector name}}", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "msg": "msg-data" - }, - "message_id": "124:191:0" - } -} -``` - -## Configuration Properties - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | yes | "" (empty string) | The AWS EventBridge [access key ID.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `secretAccessKey` | String | yes | "" (empty string) | The AWS EventBridge [secret access key.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `region` | String | yes | "" (empty string) | The region where AWS EventBridge service is located. [All AWS region](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/Region.html) | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `role` | String | false | "" (empty string) | The AWS STS [roleArn](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). Example: arn:aws:iam::598203581484:role/test-role | -| `roleSessionName` | String | false | "" (empty string) | The AWS role session name, Name it yourself. | -| `stsEndpoint` | String | false | "" (empty string) | The AWS STS endpoint. By default, the default STS endpoint: https://sts.amazonaws.com is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | false | "" (empty string) | The AWS STS region, By default, the 'region' config or env region is used. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). Example: `arn:aws:events:ap-northeast-1:598263551484:event-bus/my_eventbus` | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -> For details about this connector's advanced features and configurations, see [Advanced features](#advanced-features). - -## Advanced features - -### Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -> **Note:** -> Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the -> support of the Sink downstream system. - -### Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -### Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -### Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. - -### Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -### Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -## More Links -- [GitHub Repo](https://github.com/streamnative/pulsar-io-aws-eventbridge) -- [Announcing the Amazon EventBridge Sink Connector for Apache Pulsar](https://streamnative.io/blog/announcing-the-amazon-eventbridge-sink-connector-for-apache-pulsar) -- [Amazon EventBridge connector is now integrated with StreamNative Cloud](https://streamnative.io/blog/amazon-eventbridge-connector-is-now-integrated-with-streamnative-cloud) - - diff --git a/connectors/aws-eventbridge-sink/v3.1.0.2/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v3.1.0.2/aws-eventbridge-sink.md deleted file mode 100644 index 3e8f6989a..000000000 --- a/connectors/aws-eventbridge-sink/v3.1.0.2/aws-eventbridge-sink.md +++ /dev/null @@ -1,346 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: "/images/connectors/aws-eventbridge.png" -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: "true" -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v3.1.0.2/docs/aws-eventbridge-sink.png) - -## Quick start - -### Prerequisites - -The prerequisites for connecting an AWS EventBridge sink connector to external systems include: - -1. Create EventBridge and EventBus in AWS. -2. Create the [AWS User](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) and create `AccessKey`(Please record `AccessKey` and `SecretAccessKey`). -3. Assign permissions to AWS User, and ensure they have the following permissions to the AWS EventBus. For details, see [permissions for event buses](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-bus-perms.html) - -```json - { - "Version": "2012-10-17", - "Statement": [ - - ### To grant permissions for an account to use the PutEvents action, include the following, otherwise delete this section: ### - - { - - "Sid": "AllowAccountToPutEvents", - "Effect": "Allow", - "Principal": { - "AWS": "" - }, - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - }, - - ### Include the following section to grant permissions to all members of your AWS Organizations to use the PutEvents action ### - - { - "Sid": "AllowAllAccountsFromOrganizationToPutEvents", - "Effect": "Allow", - "Principal": "*", - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - "Condition": { - "StringEquals": { - "aws:PrincipalOrgID": "o-yourOrgID" - } - } - } - ] -} -``` -- You can set permissions directly for this user. With this method, when you create a connector, you only need to configure `accessKey` and `secretAccessKey`. -- Or you can use [Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html), this [video](https://www.youtube.com/watch?v=dqF4VJCska4) explains how to use STS on AWS. - With this method, when you create a connector, in addition to configuring `accessKey` and `secretAccessKey`, you also need to configure `role` and `roleSessionName`. - -4. Create a [Rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-rule.html) in EventBridge. -- The data structure sent to Event Bridge is described [here](## Metadata mapping), and you can create **event pattern** based on this structure. -- Set the target according to your needs. If you're testing this connector, you can set the target to [Cloud Watch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html). - - -### 1. Create a connector - -Depending on the environment, there are several ways to create an AWS EventBridge sink connector: - -- [Create Connector on StreamNative Cloud](https://docs.streamnative.io/docs/connector-create). -- [Create Connector with Function worker](https://pulsar.apache.org/docs/3.0.x/io-quickstart/). - Using this way requires you to download a **NAR** package to create a built-in or non-built-in connector. You can download the version you need from [here](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases). -- [Create Connector with Function mesh](https://functionmesh.io/docs/connectors/run-connector). - Using this way requires you to set the docker image. You can choose the version you want to launch from [here](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge/tags) - -No matter how you create an AWS EventBridge sink connector, the minimum configuration contains the following parameters. - -```json - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } -``` -> * The configuration structure varies depending on how you create the AWS EventBridge sink connector. -> For example, some are **JSON**, some are **YAML**, and some are **Kubernetes YAML**. You need to adapt the configs to the corresponding format. -> -> * If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. - -### 2. Send messages to the topic - -{% callout title="Note" type="note" %} -If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information. -{% /callout %} - -``` java - PulsarClient client = PulsarClient.builder() - .serviceUrl("{{Your Pulsar URL}}") - .build(); - - Producer producer = client.newProducer(Schema.STRING) - .topic("{{Your topic name}}") - .create(); - - String message = "{\"msg\": \"msg-data\"}"; - MessageId msgID = producer.send(message); - System.out.println("Publish " + message + " and message ID " + msgID); - - producer.flush(); - producer.close(); - client.close(); -``` - -### 3. Show data on AWS EventBridge -The connector will send the following format of JSON event to EventBridge. -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail-type": "{{Your topic name}}", - "source": "{{Your connector name}}", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "msg": "msg-data" - }, - "message_id": "124:191:0" - } -} -``` - -## Configuration Properties - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | yes | "" (empty string) | The AWS EventBridge [access key ID.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `secretAccessKey` | String | yes | "" (empty string) | The AWS EventBridge [secret access key.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `region` | String | yes | "" (empty string) | The region where AWS EventBridge service is located. [All AWS region](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/Region.html) | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `role` | String | false | "" (empty string) | The AWS STS [roleArn](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). Example: arn:aws:iam::598203581484:role/test-role | -| `roleSessionName` | String | false | "" (empty string) | The AWS role session name, Name it yourself. | -| `stsEndpoint` | String | false | "" (empty string) | The AWS STS endpoint. By default, the default STS endpoint: https://sts.amazonaws.com is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | false | "" (empty string) | The AWS STS region, By default, the 'region' config or env region is used. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). Example: `arn:aws:events:ap-northeast-1:598263551484:event-bus/my_eventbus` | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -> For details about this connector's advanced features and configurations, see [Advanced features](#advanced-features). - -## Advanced features - -### Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -{% callout title="Note" type="note" %} -Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the support of the Sink downstream system. -{% /callout %} - -### Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -### Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -### Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. - -### Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -### Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -## More Links -- [GitHub Repo](https://github.com/streamnative/pulsar-io-aws-eventbridge) -- [Announcing the Amazon EventBridge Sink Connector for Apache Pulsar](https://streamnative.io/blog/announcing-the-amazon-eventbridge-sink-connector-for-apache-pulsar) -- [Amazon EventBridge connector is now integrated with StreamNative Cloud](https://streamnative.io/blog/amazon-eventbridge-connector-is-now-integrated-with-streamnative-cloud) - - diff --git a/connectors/aws-eventbridge-sink/v3.1.0.3/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v3.1.0.3/aws-eventbridge-sink.md deleted file mode 100644 index ecffda1db..000000000 --- a/connectors/aws-eventbridge-sink/v3.1.0.3/aws-eventbridge-sink.md +++ /dev/null @@ -1,346 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: "/images/connectors/aws-eventbridge.png" -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: "true" -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v3.1.0.3/docs/aws-eventbridge-sink.png) - -## Quick start - -### Prerequisites - -The prerequisites for connecting an AWS EventBridge sink connector to external systems include: - -1. Create EventBridge and EventBus in AWS. -2. Create the [AWS User](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) and create `AccessKey`(Please record `AccessKey` and `SecretAccessKey`). -3. Assign permissions to AWS User, and ensure they have the following permissions to the AWS EventBus. For details, see [permissions for event buses](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-bus-perms.html) - -```json - { - "Version": "2012-10-17", - "Statement": [ - - ### To grant permissions for an account to use the PutEvents action, include the following, otherwise delete this section: ### - - { - - "Sid": "AllowAccountToPutEvents", - "Effect": "Allow", - "Principal": { - "AWS": "" - }, - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - }, - - ### Include the following section to grant permissions to all members of your AWS Organizations to use the PutEvents action ### - - { - "Sid": "AllowAllAccountsFromOrganizationToPutEvents", - "Effect": "Allow", - "Principal": "*", - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - "Condition": { - "StringEquals": { - "aws:PrincipalOrgID": "o-yourOrgID" - } - } - } - ] -} -``` -- You can set permissions directly for this user. With this method, when you create a connector, you only need to configure `accessKey` and `secretAccessKey`. -- Or you can use [Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html), this [video](https://www.youtube.com/watch?v=dqF4VJCska4) explains how to use STS on AWS. - With this method, when you create a connector, in addition to configuring `accessKey` and `secretAccessKey`, you also need to configure `role` and `roleSessionName`. - -4. Create a [Rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-rule.html) in EventBridge. -- The data structure sent to Event Bridge is described [here](## Metadata mapping), and you can create **event pattern** based on this structure. -- Set the target according to your needs. If you're testing this connector, you can set the target to [Cloud Watch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html). - - -### 1. Create a connector - -Depending on the environment, there are several ways to create an AWS EventBridge sink connector: - -- [Create Connector on StreamNative Cloud](https://docs.streamnative.io/docs/connector-create). -- [Create Connector with Function worker](https://pulsar.apache.org/docs/3.0.x/io-quickstart/). - Using this way requires you to download a **NAR** package to create a built-in or non-built-in connector. You can download the version you need from [here](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases). -- [Create Connector with Function mesh](https://functionmesh.io/docs/connectors/run-connector). - Using this way requires you to set the docker image. You can choose the version you want to launch from [here](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge/tags) - -No matter how you create an AWS EventBridge sink connector, the minimum configuration contains the following parameters. - -```json - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } -``` -> * The configuration structure varies depending on how you create the AWS EventBridge sink connector. -> For example, some are **JSON**, some are **YAML**, and some are **Kubernetes YAML**. You need to adapt the configs to the corresponding format. -> -> * If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. - -### 2. Send messages to the topic - -{% callout title="Note" type="note" %} -If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information. -{% /callout %} - -``` java - PulsarClient client = PulsarClient.builder() - .serviceUrl("{{Your Pulsar URL}}") - .build(); - - Producer producer = client.newProducer(Schema.STRING) - .topic("{{Your topic name}}") - .create(); - - String message = "{\"msg\": \"msg-data\"}"; - MessageId msgID = producer.send(message); - System.out.println("Publish " + message + " and message ID " + msgID); - - producer.flush(); - producer.close(); - client.close(); -``` - -### 3. Show data on AWS EventBridge -The connector will send the following format of JSON event to EventBridge. -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail-type": "{{Your topic name}}", - "source": "{{Your connector name}}", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "msg": "msg-data" - }, - "message_id": "124:191:0" - } -} -``` - -## Configuration Properties - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | yes | "" (empty string) | The AWS EventBridge [access key ID.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `secretAccessKey` | String | yes | "" (empty string) | The AWS EventBridge [secret access key.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `region` | String | yes | "" (empty string) | The region where AWS EventBridge service is located. [All AWS region](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/Region.html) | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `role` | String | false | "" (empty string) | The AWS STS [roleArn](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). Example: arn:aws:iam::598203581484:role/test-role | -| `roleSessionName` | String | false | "" (empty string) | The AWS role session name, Name it yourself. | -| `stsEndpoint` | String | false | "" (empty string) | The AWS STS endpoint. By default, the default STS endpoint: https://sts.amazonaws.com is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | false | "" (empty string) | The AWS STS region, By default, the 'region' config or env region is used. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). Example: `arn:aws:events:ap-northeast-1:598263551484:event-bus/my_eventbus` | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -> For details about this connector's advanced features and configurations, see [Advanced features](#advanced-features). - -## Advanced features - -### Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -{% callout title="Note" type="note" %} -Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the support of the Sink downstream system. -{% /callout %} - -### Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -### Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -### Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. - -### Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -### Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -## More Links -- [GitHub Repo](https://github.com/streamnative/pulsar-io-aws-eventbridge) -- [Announcing the Amazon EventBridge Sink Connector for Apache Pulsar](https://streamnative.io/blog/announcing-the-amazon-eventbridge-sink-connector-for-apache-pulsar) -- [Amazon EventBridge connector is now integrated with StreamNative Cloud](https://streamnative.io/blog/amazon-eventbridge-connector-is-now-integrated-with-streamnative-cloud) - - diff --git a/connectors/aws-eventbridge-sink/v3.1.0.4/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v3.1.0.4/aws-eventbridge-sink.md deleted file mode 100644 index 170dfd319..000000000 --- a/connectors/aws-eventbridge-sink/v3.1.0.4/aws-eventbridge-sink.md +++ /dev/null @@ -1,346 +0,0 @@ ---- -description: -author: shibd,sijie,Anonymitaet,yaalsn -contributors: shibd,sijie,Anonymitaet,yaalsn -language: Java,Shell,Dockerfile -document: -source: Private source -license: Business License -license_link: "https://github.com/streamnative/pulsar-io-aws-eventbridge/blob/master/LICENSE" -tags: -alias: AWS EventBridge Sink Connector -features: [""] -icon: "/images/connectors/aws-eventbridge.png" -download: -support: streamnative -support_link: -support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -owner_name: "streamnative" -owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: -sn_available: "true" -id: "aws-eventbridge-sink" ---- - - -The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists -data to Amazon EventBridge. - -![](https://raw.githubusercontent.com/streamnative/pulsar-io-aws-eventbridge/v3.1.0.4/docs/aws-eventbridge-sink.png) - -## Quick start - -### Prerequisites - -The prerequisites for connecting an AWS EventBridge sink connector to external systems include: - -1. Create EventBridge and EventBus in AWS. -2. Create the [AWS User](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) and create `AccessKey`(Please record `AccessKey` and `SecretAccessKey`). -3. Assign permissions to AWS User, and ensure they have the following permissions to the AWS EventBus. For details, see [permissions for event buses](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-bus-perms.html) - -```json - { - "Version": "2012-10-17", - "Statement": [ - - ### To grant permissions for an account to use the PutEvents action, include the following, otherwise delete this section: ### - - { - - "Sid": "AllowAccountToPutEvents", - "Effect": "Allow", - "Principal": { - "AWS": "" - }, - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - }, - - ### Include the following section to grant permissions to all members of your AWS Organizations to use the PutEvents action ### - - { - "Sid": "AllowAllAccountsFromOrganizationToPutEvents", - "Effect": "Allow", - "Principal": "*", - "Action": "events:PutEvents", - "Resource": "{EventBusArn}" - "Condition": { - "StringEquals": { - "aws:PrincipalOrgID": "o-yourOrgID" - } - } - } - ] -} -``` -- You can set permissions directly for this user. With this method, when you create a connector, you only need to configure `accessKey` and `secretAccessKey`. -- Or you can use [Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html), this [video](https://www.youtube.com/watch?v=dqF4VJCska4) explains how to use STS on AWS. - With this method, when you create a connector, in addition to configuring `accessKey` and `secretAccessKey`, you also need to configure `role` and `roleSessionName`. - -4. Create a [Rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-rule.html) in EventBridge. -- The data structure sent to Event Bridge is described [here](## Metadata mapping), and you can create **event pattern** based on this structure. -- Set the target according to your needs. If you're testing this connector, you can set the target to [Cloud Watch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html). - - -### 1. Create a connector - -Depending on the environment, there are several ways to create an AWS EventBridge sink connector: - -- [Create Connector on StreamNative Cloud](https://docs.streamnative.io/docs/connector-create). -- [Create Connector with Function worker](https://pulsar.apache.org/docs/3.0.x/io-quickstart/). - Using this way requires you to download a **NAR** package to create a built-in or non-built-in connector. You can download the version you need from [here](https://github.com/streamnative/pulsar-io-aws-eventbridge/releases). -- [Create Connector with Function mesh](https://functionmesh.io/docs/connectors/run-connector). - Using this way requires you to set the docker image. You can choose the version you want to launch from [here](https://hub.docker.com/r/streamnative/pulsar-io-aws-eventbridge/tags) - -No matter how you create an AWS EventBridge sink connector, the minimum configuration contains the following parameters. - -```json - "configs": { - "accessKeyId": "{{Your access access key}}", - "secretAccessKey": "{{Your secret access key}}", - "region": "test-region", - "eventBusName": "test-event-bus-name" - } -``` -> * The configuration structure varies depending on how you create the AWS EventBridge sink connector. -> For example, some are **JSON**, some are **YAML**, and some are **Kubernetes YAML**. You need to adapt the configs to the corresponding format. -> -> * If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference. - -### 2. Send messages to the topic - -{% callout title="Note" type="note" %} -If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information. -{% /callout %} - -``` java - PulsarClient client = PulsarClient.builder() - .serviceUrl("{{Your Pulsar URL}}") - .build(); - - Producer producer = client.newProducer(Schema.STRING) - .topic("{{Your topic name}}") - .create(); - - String message = "{\"msg\": \"msg-data\"}"; - MessageId msgID = producer.send(message); - System.out.println("Publish " + message + " and message ID " + msgID); - - producer.flush(); - producer.close(); - client.close(); -``` - -### 3. Show data on AWS EventBridge -The connector will send the following format of JSON event to EventBridge. -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail-type": "{{Your topic name}}", - "source": "{{Your connector name}}", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "msg": "msg-data" - }, - "message_id": "124:191:0" - } -} -``` - -## Configuration Properties - -Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the -descriptions. - -| Name | Type | Required | Default | Description | -|-------------------------|--------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `accessKeyId` | String | yes | "" (empty string) | The AWS EventBridge [access key ID.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `secretAccessKey` | String | yes | "" (empty string) | The AWS EventBridge [secret access key.](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) | -| `region` | String | yes | "" (empty string) | The region where AWS EventBridge service is located. [All AWS region](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/Region.html) | -| `eventBusName` | String | yes | "" (empty string) | The Event Bus name. | -| `role` | String | false | "" (empty string) | The AWS STS [roleArn](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). Example: arn:aws:iam::598203581484:role/test-role | -| `roleSessionName` | String | false | "" (empty string) | The AWS role session name, Name it yourself. | -| `stsEndpoint` | String | false | "" (empty string) | The AWS STS endpoint. By default, the default STS endpoint: https://sts.amazonaws.com is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. | -| `stsRegion` | String | false | "" (empty string) | The AWS STS region, By default, the 'region' config or env region is used. | -| `eventBusResourceName` | String | no | "" (empty string) | The Event Bus ARN (AWS Resource Name). Example: `arn:aws:events:ap-northeast-1:598263551484:event-bus/my_eventbus` | -| `metaDataField` | String | no | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`, `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`. | -| `batchPendingQueueSize` | int | no | 1000 | Pending queue size. This value must be greater than `batchMaxSize`. | -| `batchMaxSize` | int | no | 10 | Maximum number of batch messages. The number must be less than or equal to 10 (AWS required). | -| `batchMaxBytesSize` | long | no | 640 | Maximum number of batch bytes payload size. This value cannot be greater than 512KB. | -| `batchMaxTimeMs` | long | no | 5000 | Batch max wait time: milliseconds. | -| `maxRetryCount` | long | no | 100 | Maximum number of retries to send events, when put events failed. | -| `intervalRetryTimeMs` | long | no | 1000 | The interval time(milliseconds) for each retry, when the put events failed. | - -> For details about this connector's advanced features and configurations, see [Advanced features](#advanced-features). - -## Advanced features - -### Delivery guarantees - -The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**. - -{% callout title="Note" type="note" %} -Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the support of the Sink downstream system. -{% /callout %} - -### Data convert - -In AWS EventBridge, all events -is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html). - -Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors -recognize it and convert it to a JSON string according to the following table: - -| Pulsar Schema | Convert to JSON | Note | -|----------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| Primitive | ✔* | Just support primitive type is string and data is JSON format. | -| Avro | ✔ | Take advantage of toolkit conversions | -| Json | ✔ | Just send it directly | -| Protobuf | X | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. | -| ProtobufNative | ✔ | Take advantage of toolkit conversions | - -In EventBridge, the user data is in the `detail$data` field. - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -### Metadata mapping - -In EventBridge, a complete event contains -many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These -system fields can help you to configure the rule. - -An **Event** containing event data: - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - } - } -} -``` - -This connector maps the following fields: - -- sourceType: The default value is `${{Connector Name}}`. -- detailType: The default value is `${{Topic Name}}`. - -And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field). - -You can select the desired metadata through the following configuration: - -```jsx -# -optional: schema_version | partition | event_time | publish_time -# -message_id | sequence_id | producer_name | key | properties -metaDataField = event_time, message_id -``` - -An **Event** containing metadata : - -```json -{ - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "source-type": "test-aws-event-bridge-sink-connector", - "detail-type": "topic_name_test_1", - "source": "aws.ec2", - "account": "111122223333", - "time": "2017-12-22T18:43:48Z", - "region": "us-west-1", - "resources": [ - "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0" - ], - "detail": { - "data": { - "instance-id": " i-1234567890abcdef0", - "state": "terminated" - }, - "event_time": 789894645625, - "message_id": "1,1,1" - } -} -``` - -### Parallelism - -You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple -sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together -according to the configured subscription mode. - -Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model. - -To increase the write throughput, you can configure the following: - -```jsx -parallelism = 4 -``` - -> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used. - -### Batch Put - -AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters: - -- **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no - trigger. -- **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events. - This value should be less than 256000 and greater than 0, The default value is 640. -- **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending - events. `0` means no trigger. - -In addition to these three parameters that control flush -behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches -larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a -flush. - -### Retry Put - -In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error -message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html). - -This connector will provide two flow configs for the controller's retry strategy: - -```jsx -maxRetryCount: 100 // Maximum retry send event count, when event send failed. -intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed. -``` - -## More Links -- [GitHub Repo](https://github.com/streamnative/pulsar-io-aws-eventbridge) -- [Announcing the Amazon EventBridge Sink Connector for Apache Pulsar](https://streamnative.io/blog/announcing-the-amazon-eventbridge-sink-connector-for-apache-pulsar) -- [Amazon EventBridge connector is now integrated with StreamNative Cloud](https://streamnative.io/blog/amazon-eventbridge-connector-is-now-integrated-with-streamnative-cloud) - - diff --git a/connectors/aws-eventbridge-sink/v3.1.0.5/aws-eventbridge-sink.md b/connectors/aws-eventbridge-sink/v3.1.0.5/aws-eventbridge-sink.md index e10b4d33e..1fbd8001d 100644 --- a/connectors/aws-eventbridge-sink/v3.1.0.5/aws-eventbridge-sink.md +++ b/connectors/aws-eventbridge-sink/v3.1.0.5/aws-eventbridge-sink.md @@ -11,13 +11,13 @@ tags: alias: AWS EventBridge Sink Connector features: [""] icon: "/images/connectors/aws-eventbridge.png" -download: +download: Business License support: streamnative -support_link: +support_link: support_img: "https://avatars.githubusercontent.com/u/44651383?v=4" owner_name: "streamnative" owner_img: "https://avatars.githubusercontent.com/u/44651383?v=4" -dockerfile: +dockerfile: sn_available: "true" id: "aws-eventbridge-sink" --- diff --git a/connectors/debezium-mongodb-source/debezium-mongodb-source.yaml b/connectors/debezium-mongodb-source/debezium-mongodb-source.yaml deleted file mode 100644 index 57d31cee7..000000000 --- a/connectors/debezium-mongodb-source/debezium-mongodb-source.yaml +++ /dev/null @@ -1 +0,0 @@ -sn_available: true \ No newline at end of file diff --git a/connectors/debezium-mssql-source/debezium-mssql-source.yaml b/connectors/debezium-mssql-source/debezium-mssql-source.yaml deleted file mode 100644 index 57d31cee7..000000000 --- a/connectors/debezium-mssql-source/debezium-mssql-source.yaml +++ /dev/null @@ -1 +0,0 @@ -sn_available: true \ No newline at end of file diff --git a/connectors/debezium-mysql-source/debezium-mysql-source.yaml b/connectors/debezium-mysql-source/debezium-mysql-source.yaml deleted file mode 100644 index 57d31cee7..000000000 --- a/connectors/debezium-mysql-source/debezium-mysql-source.yaml +++ /dev/null @@ -1 +0,0 @@ -sn_available: true \ No newline at end of file diff --git a/connectors/debezium-postgres-source/debezium-postgres-source.yaml b/connectors/debezium-postgres-source/debezium-postgres-source.yaml deleted file mode 100644 index 57d31cee7..000000000 --- a/connectors/debezium-postgres-source/debezium-postgres-source.yaml +++ /dev/null @@ -1 +0,0 @@ -sn_available: true \ No newline at end of file diff --git a/connectors/elasticsearch-sink/elasticsearch-sink.yaml b/connectors/elasticsearch-sink/elasticsearch-sink.yaml deleted file mode 100644 index 57d31cee7..000000000 --- a/connectors/elasticsearch-sink/elasticsearch-sink.yaml +++ /dev/null @@ -1 +0,0 @@ -sn_available: true \ No newline at end of file diff --git a/connectors/kafka-source/kafka-source.yaml b/connectors/kafka-source/kafka-source.yaml deleted file mode 100644 index d97a0423c..000000000 --- a/connectors/kafka-source/kafka-source.yaml +++ /dev/null @@ -1,2 +0,0 @@ -repository: https://github.com/streamnative/pulsar-io-kafka -sn_available: true \ No newline at end of file diff --git a/connectors/kinesis-sink/kinesis-sink.yaml b/connectors/kinesis-sink/kinesis-sink.yaml deleted file mode 100644 index 57d31cee7..000000000 --- a/connectors/kinesis-sink/kinesis-sink.yaml +++ /dev/null @@ -1 +0,0 @@ -sn_available: true \ No newline at end of file diff --git a/hub.template b/hub.template index 4c4b8e870..8eeb2a040 100644 --- a/hub.template +++ b/hub.template @@ -4,20 +4,20 @@ author: <%= authors %> contributors: <%= contributors %> language: <%= languages %> document: -source: "<%= source %>" +source: <%= source %> license: <%= license %> license_link: "<%= licenseLink %>" tags: <%= tags %> alias: <%= alias %> features: ["<%= features %>"] icon: "<%= icon %>" -download: "<%= download %>" +download: <%= download %> support: <%= support %> support_link: <%= supportLink %> support_img: "<%= supportImg %>" owner_name: "<%= ownerName %>" owner_img: "<%= ownerImg %>" -dockerfile: "<%= dockerfile %>" +dockerfile: <%= dockerfile %> sn_available: "<%= snAvailable %>" id: "<%= id %>" --- diff --git a/sync.js b/sync.js index 5c7b3a32e..2bf6b9608 100644 --- a/sync.js +++ b/sync.js @@ -38,7 +38,7 @@ function getDocLink(organization, repository, version, fileName) { ); } -async function getTags(organization, repository) { +async function getTagsToSync(organization, repository, maxVersions) { const tagsLink = getLink(organization, repository, "git/refs/tags"); try { const { data } = await axios.get(tagsLink, { @@ -53,8 +53,29 @@ async function getTags(organization, repository) { tarball_url: tag.url.replace("/git/", "/tarball/"), }; }); - return tags.filter((tag) => { - return reThreeNumber.test(tag.name) || reFourNumber.test(tag.name); + + // 1. we need to create an object to store the maximum version for each group + let maxTagInGroup = {}; + tags.forEach(tag => { + // Check if the version format is valid + const isValidFormat = reThreeNumber.test(tag.name) || reFourNumber.test(tag.name); + if (!isValidFormat) { + return; + } + // Extract the first two digits of the version number as the group name + const group = tag.name.split('.').slice(0, 2).join('.'); + // If this group in maxTagInGroup does not have a version yet, or if this version is greater than the version in maxTagInGroup, then update maxTagInGroup + if (!maxTagInGroup[group] || compareVersions(tag.name, maxTagInGroup[group].name) > 0) { + maxTagInGroup[group] = tag; + } + }); + // 2. we convert maxTagInGroup to an array and filter out the versions that need to be synchronized + return Object.values(maxTagInGroup).filter(tag => { + // Extract the first two digits of the version number as the group name + const group = tag.name.split('.').slice(0, 2).join('.'); + + // If there is no local version for this group, or if this version is greater than the maximum local version, then synchronize this version + return !maxVersions[group] || compareVersions(tag.name, maxVersions[group]) > 0; }); } catch(error) { console.log( @@ -154,6 +175,170 @@ async function getTopics(organization, repository) { return data; } +function compareVersions(v1, v2) { + const parts1 = v1.slice(1).split('.').map(Number); + const parts2 = v2.slice(1).split('.').map(Number); + + for (let i = 0; i < parts1.length; i++) { + if (parts1[i] > parts2[i]) { + return 1; + } else if (parts1[i] < parts2[i]) { + return -1; + } + } + return 0; +} + +function getLocalMaxVersion(dir) { + const versions = fs.readdirSync(dir).filter(file => file.startsWith('v')); + const maxVersions = {}; + + versions.forEach(version => { + const group = version.split('.').slice(0, 2).join('.'); + if (!maxVersions[group] || compareVersions(version, maxVersions[group]) > 0) { + maxVersions[group] = version; + } + }); + return maxVersions; +} + +async function syncDoc(tag, pathPrefix, fileName, organization, repository, project) { + const version = tag.name; + const download = tag.tarball_url; + const _dir = pathPrefix + "/" + version; + const _file_path = _dir + "/" + fileName + ".md"; + if (!fs.existsSync(_dir)) { + fs.mkdirSync(_dir); + } + if (!fs.existsSync(_file_path)) { + const repo = await getRepository(organization, repository); + const description = repo["description"]; + const ownerName = repo["owner"]["login"]; + const support = repo["owner"]["login"]; + const ownerImg = repo["owner"]["avatar_url"]; + const icon = project.icon || ownerImg + const name = repo["name"]; + const source = repo["html_url"]; + const licenses = await getLicenses(organization, repository); + const license = licenses["license"]["name"]; + const licenseLink = licenses["html_url"]; + const languages = await getLanguages(organization, repository); + const languageKeys = Object.keys(languages); + const language = languageKeys.slice(0, 4); + const topics = await getTopics(organization, repository); + const contributors = await getContributors(organization, repository); + const contributorList = []; + for (let m = 0; m < contributors.length; m++) { + contributorList.push(contributors[m]["login"]); + } + + const doc = await getDoc( + organization, + repository, + version, + fileName + ".md" + ); + if (!doc) { + console.log(`not found doc ${fileName} version: ${version}`); + return; + } + const md = matter(doc); + const dockerfile = md.data.dockerfile; + const alias = md.data.alias || name; + let content = ""; + md.content.split("\n").forEach(function (line) { + if (line.indexOf("![](/docs") >= 0) { + line = line.replace( + "/docs", + "https://raw.githubusercontent.com/" + + organization + + "/" + + name + + "/" + + tag.name + + "/docs" + ); + } + if (line.indexOf("![](docs") >= 0) { + line = line.replace( + "docs", + "https://raw.githubusercontent.com/" + + organization + + "/" + + name + + "/" + + tag.name + + "/docs" + ); + } + content += line + "\n"; + }); + + const result = { + description: project.description || description, + authors: project.authors || contributorList.slice(0, 4), + contributors: project.contributors || contributorList.slice(0, 4), + languages: project.language || language, + document: "", + source: project.private_source ? "Private source" : (project.source || source), + license: project.private_source ? "Business License" : (project.license || license), + licenseLink: project.license_link || licenseLink, + tags: project.tags || topics["names"], + alias: project.alias || alias, + features: project.description || description, + icon: project.icon || icon, + download: project.private_source ? "Business License" : (project.download || download), + support: project.support || support, + supportLink: project.private_source ? "" : (project.support_link || source), + supportImg: project.support_img || ownerImg, + ownerName: project.owner_name || ownerName, + ownerImg: project.owner_img || ownerImg, + dockerfile: project.private_source ? "" : (project.dockerfile || dockerfile), + snAvailable: project.sn_available || md.data.sn_available, + id: fileName, + content: content, + }; + + ejs.renderFile("hub.template", result, (err, str) => { + if (err) { + return console.error(err); + } + fs.writeFileSync(_file_path, str); + console.log("successed sync doc:", _file_path); + }); + } +} + +function cleanupDirectories(baseDir) { + // Get all directories within the base directory + const dirents = fs.readdirSync(baseDir, { withFileTypes: true }); + const directories = dirents + .filter(dirent => dirent.isDirectory()) + .map(dirent => dirent.name); + + // Group the directories based on the first two digits of the version + let groups = {}; + directories.forEach(dir => { + const group = dir.split('.').slice(0, 2).join('.'); + if (!groups[group]) { + groups[group] = []; + } + groups[group].push(dir); + }); + + // For each group, delete all directories except for the maximum version + for (let group in groups) { + let versions = groups[group]; + versions.sort(compareVersions); + let maxVersion = versions.pop(); // Keep the maximum version + // Delete the other versions + for (let version of versions) { + let directoryPath = path.join(baseDir, version); + fs.rmSync(directoryPath, { recursive: true, force: true }); + } + } +} + async function fetchDocs() { const yamlFiles = await globby(yamlPatterns); for (let yamlFile of yamlFiles) { @@ -161,121 +346,27 @@ async function fetchDocs() { const fileName = path.basename(yamlFile, ".yaml"); const pathPrefix = filePath.slice(0, 2).join("/"); const project = yaml.load(fs.readFileSync(yamlFile, "utf8")); - if (!project.repository) { - continue + if (!project.repository) { + return; } const host = project.repository.split("://")[1]; const orgRepository = host.split("/"); const organization = orgRepository[1]; const repository = orgRepository[2]; - const tags = await getTags(organization, repository); - for (let tag of tags) { - const version = tag.name; - const download = tag.tarball_url; - const _dir = pathPrefix + "/" + version; - const _file_path = _dir + "/" + fileName + ".md"; - if (!fs.existsSync(_dir)) { - fs.mkdirSync(_dir); - } - if (!fs.existsSync(_file_path)) { - const repo = await getRepository(organization, repository); - const description = repo["description"]; - const ownerName = repo["owner"]["login"]; - const support = repo["owner"]["login"]; - const ownerImg = repo["owner"]["avatar_url"]; - const icon = project.icon || ownerImg - const name = repo["name"]; - const source = repo["html_url"]; - const licenses = await getLicenses(organization, repository); - const license = licenses["license"]["name"]; - const licenseLink = licenses["html_url"]; - const languages = await getLanguages(organization, repository); - const languageKeys = Object.keys(languages); - const language = languageKeys.slice(0, 4); - const topics = await getTopics(organization, repository); - const contributors = await getContributors(organization, repository); - const contributorList = []; - for (let m = 0; m < contributors.length; m++) { - contributorList.push(contributors[m]["login"]); - } - - const doc = await getDoc( - organization, - repository, - version, - fileName + ".md" - ); - if (!doc) { - console.log(`not found doc ${fileName} version: ${version}`); - continue; - } - const md = matter(doc); - const dockerfile = md.data.dockerfile; - const alias = md.data.alias || name; - let content = ""; - md.content.split("\n").forEach(function (line) { - if (line.indexOf("![](/docs") >= 0) { - line = line.replace( - "/docs", - "https://raw.githubusercontent.com/" + - organization + - "/" + - name + - "/" + - tag.name + - "/docs" - ); - } - if (line.indexOf("![](docs") >= 0) { - line = line.replace( - "docs", - "https://raw.githubusercontent.com/" + - organization + - "/" + - name + - "/" + - tag.name + - "/docs" - ); - } - content += line + "\n"; - }); - const result = { - description: project.description || description, - authors: project.authors || contributorList.slice(0, 4), - contributors: project.contributors || contributorList.slice(0, 4), - languages: project.language || language, - document: "", - source: project.private_source ? "Private source" : (project.source || source), - license: project.private_source ? "Business License" : (project.license || license), - licenseLink: project.license_link || licenseLink, - tags: project.tags || topics["names"], - alias: project.alias || alias, - features: project.description || description, - icon: project.icon || icon, - download: project.private_source ? "Business License" : (project.download || download), - support: project.support || support, - supportLink: project.private_source ? "" : (project.support_link || source), - supportImg: project.support_img || ownerImg, - ownerName: project.owner_name || ownerName, - ownerImg: project.owner_img || ownerImg, - dockerfile: project.private_source ? "" : (project.dockerfile || dockerfile), - snAvailable: project.sn_available || md.data.sn_available, - id: fileName, - content: content, - }; - - ejs.renderFile("hub.template", result, (err, str) => { - if (err) { - return console.error(err); - } - fs.writeFileSync(_file_path, str); - console.log("successed sync doc:", _file_path); - }); - } + console.log(`Initiating synchronization for ${repository} documents...`); + // 1. Get the maximum versions locally. + const maxVersions = getLocalMaxVersion(path.dirname(yamlFile)); + // 2. Get the tags (versions) that need to be synchronized. + const tags = await getTagsToSync(organization, repository, maxVersions); + // 3. Synchronize the documentation. + for (let tag of tags) { + await syncDoc(tag, pathPrefix, fileName, organization, repository, project); } + // 4. Clean up the directories right after the synchronization. + cleanupDirectories(path.dirname(yamlFile)); + console.log(`Synchronization completed for ${repository} documents.`); } } -fetchDocs(); +await fetchDocs();