Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Azure Storage: Azure storage support #3253 #3254

Merged
merged 81 commits into from
Oct 3, 2024
Merged

Azure Storage: Azure storage support #3253 #3254

merged 81 commits into from
Oct 3, 2024

Conversation

sfali
Copy link
Contributor

@sfali sfali commented Aug 27, 2024 •
edited
Loading

References #3253

Work in progress, still needs to add more tests and documentations.

Code style is copied from AWS S3 connector.

sfali added 30 commits August 23, 2024 16:55
@sfali
Setting up module for Azure storage akka#3253
06eac15
@sfali
Azure storage common settings akka#3253
42b8505
@sfali
Some common functions akka#3253
4412dec
@sfali
Supporting model classes akka#3253
dc26cea
@sfali
Exception representing returned error akka#3253
882afcd 
It parses the returned XML and populate StorageException
@sfali
Helper class to sign and populate Authorization header akka#3253
5a8728e 
1. Populates required date and version header
2. Reads current request and create authorization token as per AWS requirements
3. Populates authorization header
@sfali
Implementation of Azure storage functions akka#3253
3616c5f 
Current implementation only supports Blob and File services.
Currently supported functions:
1. Get blob and file
2. Put blob
3. Create file
3. Put range
4. Clear range
5. Delete blob and file
6. Get blob and file properties
@sfali
Scala and Java facade to implementation akka#3253
7626390
@sfali
Introducing Akka stream attributes in order to configure http request a…
f1de209 
…kka#3253
@sfali
Resolve settings from Akka stream attributes akka#3253
426b800
@sfali
Added support for SharedKeyLite authorization akka#3253
e41869e
@sfali
Added support for SAS authorization akka#3253
5d3b55d
@sfali
use option for null String akka#3253
dcc9c21
@sfali
function to get default settings akka#3253
582edb3
@sfali
Support for SAS authorization akka#3253
7dd77c6 
1. Create query string with SAS token
2. Initialize query string
@sfali
Splitting service operations based on service type akka#3253
d29f3f9
@sfali
Splitting service operations based on service type akka#3253
997746a
@sfali
StorageException test akka#3253
34832f6
@sfali
creating constants for authorization types akka#3253
6cc3254
@sfali
implicit class to handle validations akka#3253
d51c06a
@sfali
validations of configs and falling back to default values akka#3253
7b3ed57
@sfali
StorageSettings tests akka#3253
07e64a5
@sfali
support for create container operation akka#3253
44c2f84
@sfali
support for endpoint URL for local testing akka#3253
93c2aab
@sfali
using end point if defined to create URI akka#3253
7ac0aa4
@sfali
move signer out of retry flow, we don't need to sign every time akka#…
9f013cb 
…3253
@sfali
copy right header akka#3253
9b81f84
@sfali
pass implicit ActorSystem akka#3253
3ad11ea
@sfali
parse content MD5 as well akka#3253
790fd5a
@sfali
populate environment variables to run test akka#3253
d140558
sfali and others added 4 commits September 9, 2024 11:03
@sfali @girdharshubham
Update azure-storage/src/main/resources/reference.conf
67c2eaa 
Co-authored-by: Shubham Girdhar <girdharshubham@hotmail.com>
@sfali @girdharshubham
Update azure-storage/src/main/scala/akka/stream/alpakka/azure/storage…
4cfe27a 
…/StorageAttributes.scala

Co-authored-by: Shubham Girdhar <girdharshubham@hotmail.com>
@sfali @girdharshubham
Update azure-storage/src/main/scala/akka/stream/alpakka/azure/storage…
f5971a1 
…/impl/AzureStorageStream.scala

Co-authored-by: Shubham Girdhar <girdharshubham@hotmail.com>
@sfali @girdharshubham
Update azure-storage/src/main/scala/akka/stream/alpakka/azure/storage…
1eaf5df 
…/models.scala

Co-authored-by: Shubham Girdhar <girdharshubham@hotmail.com>
girdharshubham
girdharshubham reviewed Sep 9, 2024
View reviewed changes
Copy link
Contributor

@girdharshubham girdharshubham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good so far.

sfali and others added 6 commits September 10, 2024 10:45
@sfali @girdharshubham
Update azure-storage/src/main/scala/akka/stream/alpakka/azure/storage…
18f4336 
…/StorageAttributes.scala

Co-authored-by: Shubham Girdhar <girdharshubham@hotmail.com>
@sfali @girdharshubham
Update azure-storage/src/main/scala/akka/stream/alpakka/azure/storage…
fcd4326 
…/StorageAttributes.scala

Co-authored-by: Shubham Girdhar <girdharshubham@hotmail.com>
@sfali
Fail safe error message akka#3253
948db31 
1. Make sure we don't emit empty string
2. correct configuration
@sfali
Merge remote-tracking branch 'origin/3253_azure_storage' into 3253_az…
60f9ae9 
…ure_storage
@sfali
Delete Container support akka#3253
689cd53 
1. Request builder for delete container
2. Implementation of delete container and Scala and Java API
3. Tests for delete container
4. Documentation
@sfali
Create & Delete directory support akka#3253
d6d9ed0 
1. Request builder for create & delete directories
2. Implementation of create & delete directories and Scala and Java API
3. Tests for create & delete directories
4. Documentation
@sfali
Copy link
Contributor Author

sfali commented Sep 14, 2024

Issue with Java API was misconfiguration, not sure why AzuriteIntegrationSpec is failing, any idea @girdharshubham?

@girdharshubham
Copy link
Contributor

Issue with Java API was misconfiguration, not sure why AzuriteIntegrationSpec is failing, any idea @girdharshubham?

Hi @sfali
Sorry this got delayed, I will check this soon.

@sebastian-alfers
Copy link
Contributor

Hi @sfali, can you share and update on this PR? Do you want to see your contribution merged for the next release?

@sfali
Copy link
Contributor Author

sfali commented Sep 30, 2024

Hi @sfali, can you share and update on this PR? Do you want to see your contribution merged for the next release?

What do you want me to share @sebastian-alfers? The PR is failing due to couchbase and ironmq connectors are failing. not sure what should I do with these connectors.

@sebastian-alfers
Copy link
Contributor

We fixed both in main, so feel free to rebase and give it a try.

@sfali
Copy link
Contributor Author

sfali commented Sep 30, 2024

We fixed both in main, so feel free to rebase and give it a try.

@sebastian-alfers Now scaaldoc and jms connector failed. Can pin point what's going? Can you please take a look and let me know what's going on.

Thanks

@sebastian-alfers
Copy link
Contributor

Thank you for bringing this to our attention. The scaladoc failure can be ignored for now, and the jms was something we just fixed in main, so please rebase again.

@sfali
Copy link
Contributor Author

sfali commented Oct 1, 2024

Thank you for bringing this to our attention. The scaladoc failure can be ignored for now, and the jms was something we just fixed in main, so please rebase again.

Thanks @sebastian-alfers , Everything is passing other than scaaldoc

@sebastian-alfers
Copy link
Contributor

Please let me know once you want to merge it.

@sfali
Copy link
Contributor Author

sfali commented Oct 2, 2024

Please let me know once you want to merge it.

This can be merge anytime. It needs to be approved.

@sfali
Copy link
Contributor Author

sfali commented Oct 3, 2024 •
edited
Loading

It seems scaladoc pipeline needs to be successful before it can be merged. @sebastian-alfers

@sebastian-alfers sebastian-alfers merged commit 4049dca into akka:main Oct 3, 2024
47 of 48 checks passed
@sfali sfali deleted the 3253_azure_storage branch October 3, 2024 17:37
Susmit07
Susmit07 reviewed Nov 7, 2024
View reviewed changes
azure-storage/src/main/scala/akka/stream/alpakka/azure/storage/impl/AzureStorageStream.scala
objectPath = objectPath,
requestBuilder = requestBuilder)

private[storage] def putBlob(objectPath: String,
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sfali one question for large files may be of > 5-10Gb .. S3 has multipartUploadAPI for azure blob how are we going to achieve it ?

splitting in chunks?

adding some UTs would have been helpful..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Susmit07 Susmit07 Susmit07 left review comments

@girdharshubham girdharshubham girdharshubham left review comments

@sebastian-alfers sebastian-alfers sebastian-alfers approved these changes

Assignees
No one assigned
Labels
dependency-change For PRs changing the version of a dependency. documentation
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@sfali @girdharshubham @sebastian-alfers @Susmit07