Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[HP-1539] Add AWS functionality to cirrus #96

Merged
merged 21 commits into from
Aug 26, 2024
Merged

Conversation

MichaelLukowski
Copy link
Member

New Features

  • Added new functionality for AWS s3 API calls. This is for getting a presigned url for uploading a file, downloading a file, and downloading a file from a requester pays bucket

Copy link

The style in this PR agrees with black. ✔️

This formatting comment was generated automatically by a script in uc-cdis/wool.

Copy link
Collaborator

@Avantol13 Avantol13 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks pretty good so far. See comments for minor things. Make sure you update the version in the pyproject.toml too - and then send me the Fence PR using this branch and we'll wait to merge this until we can verify everything is working in updated Fence

README.md Outdated Show resolved Hide resolved
import backoff
import boto3
from botocore.exceptions import ClientError
from gen3cirrus.config import config
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer if we break up the imports by newlines for 3 sections:

  • built in libs
  • external deps
  • internal imports

Generic Amazon servicing using Boto3
"""

def __init__(self, client):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe just so it's super clear we should call this boto3_client?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bump

gen3cirrus/aws/services.py Outdated Show resolved Hide resolved
gen3cirrus/aws/utils.py Outdated Show resolved Hide resolved
pyproject.toml Outdated Show resolved Hide resolved
test/test_utils.py Outdated Show resolved Hide resolved
test/test_utils.py Show resolved Hide resolved
@mfshao mfshao requested a review from Avantol13 August 21, 2024 16:36
gen3cirrus/aws/services.py Outdated Show resolved Hide resolved
gen3cirrus/aws/utils.py Outdated Show resolved Hide resolved
gen3cirrus/aws/manager.py Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
Generic Amazon servicing using Boto3
"""

def __init__(self, client):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bump

gen3cirrus/aws/services.py Outdated Show resolved Hide resolved
gen3cirrus/aws/utils.py Outdated Show resolved Hide resolved
gen3cirrus/aws/utils.py Outdated Show resolved Hide resolved
gen3cirrus/aws/utils.py Outdated Show resolved Hide resolved
test/test_utils.py Outdated Show resolved Hide resolved
@mfshao mfshao requested a review from Avantol13 August 22, 2024 14:39
Test that we can get a presigned url from a bucket
"""

s3 = boto3.client("s3", aws_access_key_id="", aws_secret_access_key="")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not seeing where this is mocked, so I'm actually not sure how this is passing

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure it need mocks 🤔 the generate_presigned_url() in boto3 is a local operation

assert url != None


def test_aws_get_presigned_url_requester_pays():
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need more tests for the negative cases where boto throws an error

@mfshao mfshao requested a review from Avantol13 August 22, 2024 16:52
Avantol13
Avantol13 previously approved these changes Aug 22, 2024
@mfshao
Copy link
Contributor

mfshao commented Aug 22, 2024

@mfshao mfshao requested a review from Avantol13 August 26, 2024 19:10
@mfshao mfshao merged commit a0d311b into master Aug 26, 2024
7 checks passed
@mfshao mfshao deleted the feat/s3-functionality branch August 26, 2024 20:56
@mfshao mfshao mentioned this pull request Aug 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants