[FLINK-26570][statefun] Remote module configuration interpolation #309
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -61,3 +61,36 @@ spec: | |
|
|
||
| A module YAML file can contain multiple YAML documents, separated by `---`, each representing a component to be included in the application. | ||
| Each component is defined by a kind typename string and a spec object containing the component's properties. | ||
|
|
||
| # Configuration string interpolation | ||
| You can use `${placeholders}` inside `spec` elements. These will be replaced by entries from a configuration map, consisting of: | ||
| 1. System properties | ||
| 2. Environment variables | ||
| 3. flink-conf.yaml entries with prefix 'statefun.module.global-config.' | ||
| 4. Command line args | ||
|
|
||
| where (4) override (3) which override (2) which override (1). | ||
|
|
||
| Example: | ||
| ```yaml | ||
| kind: io.statefun.endpoints.v2/http | ||
| spec: | ||
| functions: com.example/* | ||
| urlPathTemplate: ${FUNC_PROTOCOL}://${FUNC_DNS}/{function.name} | ||
| --- | ||
| kind: io.statefun.kafka.v1/ingress | ||
| spec: | ||
| id: com.example/my-ingress | ||
| address: ${KAFKA_ADDRESS}:${KAFKA_PORT} | ||
| consumerGroupId: my-consumer-group | ||
| topics: | ||
| - topic: ${KAFKA_INGRESS_TOPIC} | ||
| (...) | ||
| properties: | ||
| - ssl.truststore.location: ${SSL_TRUSTSTORE_LOCATION} | ||
| - ssl.truststore.password: ${SSL_TRUSTSTORE_PASSWORD} | ||
| (...) | ||
|
Comment on lines
+76
to
+92
There was a problem hiding this comment. I really like the flexibility the templating mechanism gives to the user. The thing I am asking myself is whether it gives too much power so that the user can shoot himself into his foot. The danger I see is that we add level of indirections that make it harder to reason about what the effective yaml will look like for a user. The underlying problem the issue wants to solve is to pass in information that is only available to the process that runs SF but not the user (e.g. secrets). I am wondering whether there is an alternative to achieve the same but with a bit less power (e.g. allowing substitution only for selected fields (also confusing)). I don't have a perfect answer here. I mainly wanted to hear your and @igalshilman's opinion here. Maybe one idea could be to log the effective/resolved yaml somewhere so that the user sees what is actually run, if this does not happen already. There was a problem hiding this comment. I like the idea of logging the effective yaml somewhere. We should probably make it an opt-in kind of a deal, since we don't want to be automatically logging secrets. I'll hold off on coding this until we hear from @igalshilman |
||
| ``` | ||
| {{< hint info >}} | ||
| Please note that `{function.name}` is not a placeholder to be replaced by entries from the merged configuration. See [url template]({{< ref "docs/modules/http-endpoint" >}}) | ||
| {{< /hint >}} | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it is more common to give env variables precedence over
flink-conf.yamlvalues.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree in principal. That said, because
globalConfigurationis already an combination of args and flink-conf.yaml entries with thestatefun.module.global-config.prefix, there's no easy way to put env variables in between them without affecting other parts of the system.I made a start in my fork and the number of changes is pretty high for what we gain. Please have a look and let me know if I Should working on the change you mentioned in this comment
FilKarnicki@02cd6a9