Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: FEEL function for invoking outbound connectors #2744

Closed

Conversation

chillleader
Copy link
Member

@chillleader chillleader commented Jun 12, 2024

Description

This PR implements:

  • the ExternalConnectorExecutor that allows invoking outbound connectors outside of the job worker context
  • the ConnectorInvocationFeelFunctionProvider that can be dynamically added to the FeelEngineWrapper (e.g. as a Spring bean)
  • the TransientDataStore to store the binary data references away from serialization/FEEL engine

Related issues

closes https://github.com/camunda/team-connectors/issues/785

@chillleader chillleader self-assigned this Jun 12, 2024
@chillleader chillleader requested review from sbuettner, jonathanlukas and johnBgood and removed request for jonathanlukas June 12, 2024 07:07
@chillleader chillleader marked this pull request as ready for review June 12, 2024 17:01
@chillleader chillleader requested a review from a team as a code owner June 12, 2024 17:01
Comment on lines +45 to +50
String variablesJson;
try {
variablesJson = objectMapper.writeValueAsString(variables);
} catch (Exception e) {
throw new RuntimeException("Error while serializing variables", e);
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we transform the variables to json wont we have an issue with passing data around like a byte[] for example?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have a specific scenario in mind? As I'm thinking about it, it seems that we will be fine if we:

  • only pass binary data as part of the Document class, and
  • have a proper serialization/deserialization mechanism in place for Document that converts it to base64.

I might be missing something, and it is possible that we can do secret replacement more efficiently 🤔

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chillleader Sure, lets assume we want to execute an S3 connector in the resultExpression of the REST Connector to store the binary body. Passing a byte[] document wouldnt really be efficiently possible if we serialize it to json first and read it back again for larger files. Here it would be better to be able to directly invoke the connector passing the objects/references in memory to be able to use streams for example.

Yes, secret replacing would be an issue but this could theoretically be done by only replacing it when an object in the variables is of type string before passing it to the Connector.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for clarifying, makes sense. This requires refactoring how we replace secrets in all outbound connectors to avoid having two secret replacement methods (we are still getting a json string from the zeebe job). I'm not opposed to that and can also attempt this as part of this PR.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, you can also add secret support later on to figure out whether our approach here is reasonable 😄

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed it here :)

Copy link
Collaborator

@johnBgood johnBgood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧨

@chillleader chillleader requested a review from sbuettner June 14, 2024 16:09
Comment on lines 31 to 32
Object jsonVariables) {
super(secretProvider, validationProvider, objectMapper, jsonVariables);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we rename jsonVariables to variables?

@chillleader chillleader marked this pull request as draft June 24, 2024 07:17
@johnBgood johnBgood force-pushed the epic-1748-doc-ref-handling branch from 0c3a8bb to 381da62 Compare July 17, 2024 13:05
@chillleader
Copy link
Member Author

Closing this as per our decision to move along with a different approach.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants