Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade KCL Java dependency from 2.4.4 to 2.5.1 and Added streamArn support in KCL Python #221

Merged
merged 4 commits into from
Jun 29, 2023

Conversation

pelaezryan
Copy link
Contributor

@pelaezryan pelaezryan commented Jun 28, 2023

Description of changes:

  • Updated Python sample.properties to support streamArn. This is an extension of the KCL Java - Multilang PR to support parsing streamName from streamArn.
  • Upgraded KCL dependencies

Testing

Scenario #0 - Existing Functionality - streamName set and streamArn propert DNE

Config:

...
# over STDIN and STDOUT according to the multi-language protocol.
executableName = sample_kclpy_app.py

# The name of an Amazon Kinesis stream to process.
# Important: streamArn takes precedence over streamName if both are set
streamName = words

Output

2023-06-26 11:48:07,056 [main] INFO  s.a.k.m.MultiLangDaemonConfig [NONE] - Running PythonKCLSample to process stream words with executable sample_kclpy_app.py 

Scenario #1 - streamName and streamArn passed in. KCL uses streamArn

Config:

# The Stream arn: arn:aws:kinesis:<region>:<account id>:stream/<stream name>
# Important: streamArn takes precedence over streamName if both are set
streamArn = arn:aws:kinesis:us-east-5:000000000000:stream/words

# The name of an Amazon Kinesis stream to process.
# Important: streamArn takes precedence over streamName if both are set
streamName = ignoreThisStream

Output

39:03,298 [main] INFO  s.a.k.m.MultiLangDaemonConfig [NONE] - Running PythonKCLSample to process stream words with executable sample_kclpy_app.py

Scenario #2 - Only streamArn is passed in

Config

# The Stream arn: arn:aws:kinesis:<region>:<account id>:stream/<stream name>
# Important: streamArn takes precedence over streamName if both are set
streamArn = arn:aws:kinesis:us-east-5:000000000000:stream/words

# The name of an Amazon Kinesis stream to process.
# Important: streamArn takes precedence over streamName if both are set
streamName = 

Ouput

2023-06-26 11:43:15,536 [main] INFO  s.a.k.m.MultiLangDaemonConfig [NONE] - Running PythonKCLSample to process stream words with executable sample_kclpy_app.py 

Scenario #3 - Only streamName is passed in

Config

# The Stream arn: arn:aws:kinesis:<region>:<account id>:stream/<stream name>
# Important: streamArn takes precedence over streamName if both are set
streamArn = 

# The name of an Amazon Kinesis stream to process.
# Important: streamArn takes precedence over streamName if both are set
streamName = words

Output

2023-06-26 11:44:17,034 [main] INFO  s.a.k.m.MultiLangDaemonConfig [NONE] - Running PythonKCLSample to process stream words with executable sample_kclpy_app.py 

Scenario #4 - Neither are set

Config:

# The Stream arn: arn:aws:kinesis:<region>:<account id>:stream/<stream name>
# Important: streamArn takes precedence over streamName if both are set
streamArn = 

# The name of an Amazon Kinesis stream to process.
# Important: streamArn takes precedence over streamName if both are set
streamName = 

Output:

java.lang.IllegalArgumentException: Stream name or Stream Arn is required. Stream Arn takes precedence if both are passed in.
        at org.apache.commons.lang3.Validate.notBlank(Validate.java:441)
        at software.amazon.kinesis.multilang.config.KinesisClientLibConfigurator.getConfiguration(KinesisClientLibConfigurator.java:84)
        at software.amazon.kinesis.multilang.MultiLangDaemonConfig.<init>(MultiLangDaemonConfig.java:108)
        at software.amazon.kinesis.multilang.MultiLangDaemonConfig.<init>(MultiLangDaemonConfig.java:81)
        at software.amazon.kinesis.multilang.MultiLangDaemonConfig.<init>(MultiLangDaemonConfig.java:65)
        at software.amazon.kinesis.multilang.MultiLangDaemon.buildMultiLangDaemonConfig(MultiLangDaemon.java:171)
        at software.amazon.kinesis.multilang.MultiLangDaemon.main(MultiLangDaemon.java:220)
Stream name or Stream Arn is required. Stream Arn takes precedence if both are passed in.
Usage: amazon-kinesis-client MultiLangDaemon [options]
  Options:
    -l, --log-configuration
      File location of logback.xml to be override the default
    -p, --properties-file
      Properties file to be used with the KCL

==========

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@pelaezryan pelaezryan changed the title Added streamArn support in KCL Python Upgrade KCL Java dependency to 2.5.1 and Added streamArn support in KCL Python Jun 29, 2023
@pelaezryan pelaezryan changed the title Upgrade KCL Java dependency to 2.5.1 and Added streamArn support in KCL Python Upgrade KCL Java dependency from 2.4.4 to 2.5.1 and Added streamArn support in KCL Python Jun 29, 2023
@@ -135,6 +135,11 @@
<artifactId>apache-client</artifactId>
<version>${awssdk.version}</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>arns</artifactId>

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume that this is where the new functionality comes in to read in the arn. I also assume that it is plural because it can be a list. Is there anywhere where we document clearly that this is the case

Copy link
Contributor Author

@pelaezryan pelaezryan Jun 29, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Correct, this dependency is added for streamArn support.

  2. It is not plural because it can be a list. That is what the artifact is called (and was named by whomever is the creator of software.amazon.awssdk). It refers to the arns package: https://central.sonatype.com/artifact/software.amazon.awssdk/arns

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any way we can document this use case. I am confused on how this will surface well in the code to be documented for future developers and users. For example can we comment that link into the pom file?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maven's central repository is the link i provided above. the Pom.xml pulls dependencies from there. Therefore it's already implied that the link above refers to the dependency in the XML. an explicit link should not be needed.

@pelaezryan pelaezryan merged commit aa42159 into awslabs:master Jun 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants