Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor S3 API to static factory methods #1395

Merged
merged 41 commits into from
Jan 16, 2019
Merged

Conversation

2m
Copy link
Member

@2m 2m commented Dec 21, 2018

This is a second attempt to clean up S3 connector API. Instead of creating instances of S3Client that then are used to create Source/Flow/Sinks, all of the factories are moved to the S3 object. If connector implementation needs an ActorSystem, ExecutionContext or S3Settings, all of those are taken from the materializer.

Access to the materialized is factored out to MaterializerAccess.

This PR also enables MinioS3IntegrationSpec and configures minio docker image for that test.

Continuation of #1323

Two test cases are failing. Therefore this is still WIP. One testcase succeed uploading an empty file is tricky to fix with the current implementation of MaterializerAccess.source, as empty file (empty source) does not work as other sources with Source.lazily.

Copy link
Member

@ennru ennru left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!

  • Do a KazeClass with S3Settings
  • Why are there different models in javadsl and scaladsl? They seem to be the same. Check if they may use the WriteMessage/WriteResult naming standard.
  • In HttpRequests: the settings could be passed explicitly

I'll need another look.

@2m
Copy link
Member Author

2m commented Jan 9, 2019

Addresses PR feedback. Also updated the way Materializer and Attributes are exposed. Instead of using lazy{Source,Flow,Sink} factories, Setup.{source,flow,sink} factories are introduced which wrap given graph into a custom GraphStage.

Materialized value is still wrapped into a Future, but the problem with empty sources, which lazySink had, is gone.

Still needs a docs rewrite.

@2m 2m force-pushed the wip-mat-from-gi-2m branch 2 times, most recently from 377ff4b to 33f67fc Compare January 14, 2019 12:46
@2m 2m changed the title WIP Refactor S3 API to static factory methods Refactor S3 API to static factory methods Jan 14, 2019
@2m
Copy link
Member Author

2m commented Jan 14, 2019

Docs have been refactored. The IBM Bluemix section has been moved to a separate page. Also added a new paragraph and code snipped about using S3Atributes.

Copy link
Member

@ennru ennru left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really like the outcome from all the work you put into this.
LGTM.

docs/src/main/paradox/s3.md Outdated Show resolved Hide resolved
s3/src/main/resources/reference.conf Outdated Show resolved Hide resolved
docs/src/main/paradox/release-notes/index.md Show resolved Hide resolved
docs/src/main/paradox/s3.md Show resolved Hide resolved
docs/src/main/paradox/release-notes/1.0-M2.md Show resolved Hide resolved
@ennru
Copy link
Member

ennru commented Jan 14, 2019

Fixes #1086

@ennru
Copy link
Member

ennru commented Jan 14, 2019

Fixes #938

@2m 2m force-pushed the wip-mat-from-gi-2m branch from 92e5e42 to dbe9f50 Compare January 15, 2019 07:54
@2m
Copy link
Member Author

2m commented Jan 15, 2019

Addressed feedback and rebased on top of the current master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants