TST: Streaming of S3 files #22520
TST: Streaming of S3 files #22520
Conversation
|
CI will crash and burn as I didn't add |
|
I think the easiest thing to do is to ensure the test is skipped without botocore, and rely on |
ci/environment-dev.yaml
Outdated
| @@ -9,6 +9,7 @@ dependencies: | |||
| - flake8-comprehensions | |||
| - hypothesis>=3.58.0 | |||
| - moto | |||
| - botocore>=1.10.47 | |||
There was a problem hiding this comment.
So this can be removed.
pandas/tests/io/test_s3.py
Outdated
| @@ -1,3 +1,7 @@ | |||
| from six import BytesIO | |||
|
|
|||
| from botocore.response import StreamingBody | |||
pandas/tests/io/test_s3.py
Outdated
|
|
||
|
|
||
| def test_streaming_s3_objects(): | ||
| data = [ |
There was a problem hiding this comment.
Then here you can do botocore_response = pytest.importorskip('botocore.response', minversion='1.10.47')
There was a problem hiding this comment.
Can we do this when the version pertains to the base module, not the imported class? I just tried it in a dummy file and all I got was Skipped: module 'botocore.response' has __version__ None, required is: '1.10.47'. So should I do something like this?
import pytest
def my_test_fn():
pytest.importorskip('botocore', minversion='1.10.47')
from botocore.response import StreamingBody
# test
There was a problem hiding this comment.
yeah, your suggestion should work.
pandas/tests/io/test_s3.py
Outdated
| @@ -1,3 +1,7 @@ | |||
| from six import BytesIO | |||
There was a problem hiding this comment.
from pandas.compat import BytesIO
Codecov Report
@@ Coverage Diff @@
## master #22520 +/- ##
=========================================
Coverage ? 92.03%
=========================================
Files ? 169
Lines ? 50780
Branches ? 0
=========================================
Hits ? 46737
Misses ? 4043
Partials ? 0
Continue to review full report at Codecov.
|
gfyoung
added
Testing
pandas/tests/io/test_s3.py
Outdated
|
|
||
|
|
||
| def test_streaming_s3_objects(): | ||
| pytest.importorskip('botocore', minversion='1.10.47') |
There was a problem hiding this comment.
can you add the issue number here as a comment
Codecov Report
@@ Coverage Diff @@
## master #22520 +/- ##
=========================================
Coverage ? 92.03%
=========================================
Files ? 169
Lines ? 50780
Branches ? 0
=========================================
Hits ? 46737
Misses ? 4043
Partials ? 0
Continue to review full report at Codecov.
|
|
thanks! |
Issue #17135 was resolved by an upstream change, this is just a unit test to avoid regressions.