Skip to content
This repository has been archived by the owner on Aug 26, 2024. It is now read-only.

Commit

Permalink
Update docs for configuring a transform and custom partitioner
Browse files Browse the repository at this point in the history
  • Loading branch information
josephcummings committed Feb 14, 2024
1 parent 3467094 commit 3cdb18e
Show file tree
Hide file tree
Showing 5 changed files with 14 additions and 14 deletions.
4 changes: 2 additions & 2 deletions content/docs/Configuration/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Available configuration options are:
| `replicator.sink.connectionString` | Connection string for the target cluster or instance |
| `replicator.sink.protocol` | Writer protocol (`tcp` or `grpc`) |
| `replicator.sink.partitionCount` | Number of [partitioned]({{% ref "writers" %}}) concurrent writers |
| `replicator.sink.partitioner.file` | Custom JavaScript [partitioner]({{% ref "writers" %}}) |
| `replicator.sink.partitioner` | Custom JavaScript [partitioner]({{% ref "writers" %}}) |
| `replicator.sink.bufferSize` | Size of the sink buffer, `1000` events by default |
| `replicator.scavenge` | Enable real-time [scavenge]({{% ref "scavenge" %}}) |
| `replicator.runContinuously` | Set to `false` if you want Replicator to stop when it reaches the end of `$all` stream. Default is `true`, so the replication continues until you stop it explicitly. |
Expand Down Expand Up @@ -53,7 +53,7 @@ replicator:
partitionCount: 1
transform:
type: http
endpoint: https://my.acme.org/transform
config: https://my.acme.org/transform
scavenge: false
filters: []
checkpoint:
Expand Down
9 changes: 5 additions & 4 deletions content/docs/Deployment/Kubernetes/helm.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ replicator:
exclude: "((Bad|Wrong)\w+Event)"
transform:
type: http
endpoint: "http://transform.somenamespace.svc:5000"
config: "http://transform.somenamespace.svc:5000"
prometheus:
metrics: true
operator: true
Expand All @@ -60,7 +60,7 @@ Available options are:
| `replicator.sink.connectionString` | Connection string for the target cluster or instance | nil |
| `replicator.sink.protocol` | Writer protocol | `grpc` |
| `replicator.sink.partitionCount` | Number of [partitioned]({{% ref "writers" %}}) concurrent writers | `1` |
| `replicator.sink.partitioner.file` | Custom JavaScript [partitioner]({{% ref "writers" %}}) | `null` |
| `replicator.sink.partitioner` | Custom JavaScript [partitioner]({{% ref "writers" %}}) | `null` |
| `replicator.sink.bufferSize` | Size of the sink buffer, in events | `1000` |
| `replicator.scavenge` | Enable real-time [scavenge]({{% ref "scavenge" %}}) | `true` |
| `replicator.runContinuously` | Set to `false` if you want Replicator to stop when it reaches the end of `$all` stream. | `true` |
Expand All @@ -75,6 +75,7 @@ Available options are:
| `resources.limits.memory` | Memory limit | `1Gi` |
| `pvc.storageClass` | Persistent volume storage class name | `null` |
| `terminationGracePeriodSeconds` | Timeout for the workload graceful shutdown, it must be long enough for the sink buffer to flush | `300` |
| `jsConfigMaps` | List of existing config maps to be used as JS code files (for JS transform, for example) | `{}` |

{{< alert title="Note:" >}}
- As Replicator uses 20.10 TCP client, you have to specify `UseSsl=false` in the connection string when connecting to an insecure cluster or instance.
Expand All @@ -93,7 +94,7 @@ Follow the documentation to configure a [JavaScript transform](/docs/features/tr

Then append the following option to your `helm install` command:
```bash
--set-file replicator.transform.js=./transform.js
--set-file transformJs=./transform.js
```

## Configuring a custom partitioner
Expand All @@ -102,7 +103,7 @@ Follow the documentation to configure a custom [partitioner]({{% ref "writers" %

Then append the following option to your `helm install` command:
```bash
--set-file replicator.partitioner.js=./partitioner.js
--set-file partitionerJs=./partitioner.js
```

## Complete the deployment
Expand Down
7 changes: 3 additions & 4 deletions content/docs/Features/Sinks/writers.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,14 +56,13 @@ There are two modes for custom partitions, described below.

As with the stream name partitioning, the custom partition key is hashed, and the hash of the key is used to decide which partition will take the event. This method allows having less partitions than there are keys.

To use this mode you need to set the partition count using the `replicator.sink.partitionCount` setting, and also specify the file name of the partitioning function in the `replicator.sink.partitioner.file` setting. For example:
To use this mode you need to set the partition count using the `replicator.sink.partitionCount` setting, and also specify the file name of the partitioning function in the `replicator.sink.partitioner` setting. For example:

```yaml
replicator:
sink:
partitionCount: 10
partitioner:
file: ./partitioner.js
partitioner: ./partitioner.js
```
### Partition by value
Expand All @@ -72,7 +71,7 @@ In some cases, it's better to assign a single partition for each partition key.
To use value-based partitioning, use the same partitioning function signature. The difference is that for each returned partition key there will be a separate partition. For example, if the function deterministically return 10 different values, there will be 10 partitions. You don't need to configure the partition count, partitions will be dynamically created based on the number of unique keys.
The settings file, therefore, only needs the `replicator.sink.partitioner.file` setting configured.
The settings file, therefore, only needs the `replicator.sink.partitioner` setting configured.

## Partition count considerations

Expand Down
4 changes: 2 additions & 2 deletions content/docs/Features/Transforms/http.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,15 +23,15 @@ Before using the HTTP transformation, you need to build and deploy the transform

The transformation configuration has two parameters:
- `replicator.transform.type` - should be `http` for HTTP transform
- `replicator.transform.endpoint` - the HTTP(S) endpoint URL
- `replicator.transform.config` - the HTTP(S) endpoint URL

For example:

```yaml
replicator:
transform:
type: http
endpoint: http://transform-func.myapp.svc.cluster.local
config: http://transform-func.myapp.svc.cluster.local
```
Replicator doesn't support any authentication, so the endpoint must be open and accessible. You can host it at the same place as Replicator itself to avoid the network latency, or elsewhere. For example, your transformation service can be running in the same Kubernetes cluster, so you can provide the internal DNS name to its service. Alternatively, you can use a serverless function.
Expand Down
4 changes: 2 additions & 2 deletions content/docs/Features/Transforms/js.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,15 +22,15 @@ For this transform, you need to supply a code snippet, written in JavaScript, wh
You can configure Replicator to use a JavaScript transformation function using the following parameters:

- `replicator.transform.type` - must be set to `js`
- `replicator.transform.file` - name of the file, which contains the transformation function
- `replicator.transform.config` - name of the file, which contains the transformation function

For example:

```yaml
replicator:
transform:
type: js
file: ./transform.js
config: ./transform.js
```
The function itself must be named `transform`. Replicator will call it with the following arguments:
Expand Down

0 comments on commit 3cdb18e

Please sign in to comment.